2023-11-15T06:21:56.341 INFO:root:teuthology version: 0.0.1.dev203+g6899cd2 2023-11-15T06:21:56.342 DEBUG:teuthology.run:Teuthology command: teuthology --archive /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622 --name yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi --verbose --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} -- /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622/orig.config.yaml 2023-11-15T06:21:56.398 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T06:21:56.487 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622 branch: wip-yuri4-testing-2023-11-13-0820-pacific description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: yweinste@redhat.com first_in_suite: false job_id: '7458622' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: wip-yuri4-testing-2023-11-13-0820-pacific ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: 620f25c1794f62207424fc9319427943fb8d1fae ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: 620f25c1794f62207424fc9319427943fb8d1fae selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: wip-yuri4-testing-2023-11-13-0820-pacific sha1: 620f25c1794f62207424fc9319427943fb8d1fae owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 4886 sha1: 620f25c1794f62207424fc9319427943fb8d1fae sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: wip-yuri4-testing-2023-11-13-0820-pacific suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 620f25c1794f62207424fc9319427943fb8d1fae targets: smithi123.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArzQg8bBU/6MM3lFQxUyLexbvAq4wWJumoQgZ4oJUVF4GWiBYIEpFJtqLXsgo4JfWOst+BCXkyuhGOolf+pGks= smithi135.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQq44W7J6KrofH3PchCIcL9iFw+bUWcuxZtbLws7vGqJip8dd4tXegIuIZuE2K8bdDTUA8GzDYV1NHrSIP45QA= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 2 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay false - cephadm.shell: host.a: - ceph fs set cephfs inline_data true --yes-i-really-really-mean-it - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: 6899cd26fceddb2fec83dc1a1349394b28c8998e timestamp: 2023-11-14_20:45:49 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.3590714 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2023-11-15T06:21:56.488 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa; will attempt to use it 2023-11-15T06:21:56.488 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks 2023-11-15T06:21:56.489 INFO:teuthology.run_tasks:Running task internal.check_packages... 2023-11-15T06:21:56.491 INFO:teuthology.task.internal:Checking packages... 2023-11-15T06:21:56.516 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash '620f25c1794f62207424fc9319427943fb8d1fae' 2023-11-15T06:21:56.517 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2023-11-15T06:21:56.517 INFO:teuthology.packaging:ref: None 2023-11-15T06:21:56.517 INFO:teuthology.packaging:tag: None 2023-11-15T06:21:56.518 INFO:teuthology.packaging:branch: wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T06:21:56.518 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:21:56.518 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=wip-yuri4-testing-2023-11-13-0820-pacific 2023-11-15T06:21:56.740 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-170.g620f25c1 2023-11-15T06:21:56.741 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2023-11-15T06:21:56.747 INFO:teuthology.task.internal:no buildpackages task found 2023-11-15T06:21:56.747 INFO:teuthology.run_tasks:Running task internal.save_config... 2023-11-15T06:21:56.798 INFO:teuthology.task.internal:Saving configuration 2023-11-15T06:21:56.818 INFO:teuthology.run_tasks:Running task internal.check_lock... 2023-11-15T06:21:56.824 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:21:56.848 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:16:39.760724', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArzQg8bBU/6MM3lFQxUyLexbvAq4wWJumoQgZ4oJUVF4GWiBYIEpFJtqLXsgo4JfWOst+BCXkyuhGOolf+pGks='} 2023-11-15T06:21:56.873 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:16:39.762082', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQq44W7J6KrofH3PchCIcL9iFw+bUWcuxZtbLws7vGqJip8dd4tXegIuIZuE2K8bdDTUA8GzDYV1NHrSIP45QA='} 2023-11-15T06:21:56.874 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2023-11-15T06:21:56.880 INFO:teuthology.task.internal:roles: ubuntu@smithi123.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2023-11-15T06:21:56.880 INFO:teuthology.task.internal:roles: ubuntu@smithi135.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2023-11-15T06:21:56.880 INFO:teuthology.run_tasks:Running task console_log... 2023-11-15T06:21:56.953 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7fe61c68eb80>, signals=[15]) 2023-11-15T06:21:56.955 INFO:teuthology.run_tasks:Running task internal.connect... 2023-11-15T06:21:56.961 INFO:teuthology.task.internal:Opening connections... 2023-11-15T06:21:56.962 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi123.front.sepia.ceph.com 2023-11-15T06:21:56.964 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:21:57.034 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi135.front.sepia.ceph.com 2023-11-15T06:21:57.035 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:21:57.104 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2023-11-15T06:21:57.114 DEBUG:teuthology.orchestra.run.smithi123:> uname -m 2023-11-15T06:21:57.138 INFO:teuthology.orchestra.run.smithi123.stdout:x86_64 2023-11-15T06:21:57.139 DEBUG:teuthology.orchestra.run.smithi123:> cat /etc/os-release 2023-11-15T06:21:57.201 INFO:teuthology.orchestra.run.smithi123.stdout:NAME="CentOS Stream" 2023-11-15T06:21:57.201 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION="8" 2023-11-15T06:21:57.201 INFO:teuthology.orchestra.run.smithi123.stdout:ID="centos" 2023-11-15T06:21:57.202 INFO:teuthology.orchestra.run.smithi123.stdout:ID_LIKE="rhel fedora" 2023-11-15T06:21:57.202 INFO:teuthology.orchestra.run.smithi123.stdout:VERSION_ID="8" 2023-11-15T06:21:57.202 INFO:teuthology.orchestra.run.smithi123.stdout:PLATFORM_ID="platform:el8" 2023-11-15T06:21:57.202 INFO:teuthology.orchestra.run.smithi123.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T06:21:57.202 INFO:teuthology.orchestra.run.smithi123.stdout:ANSI_COLOR="0;31" 2023-11-15T06:21:57.203 INFO:teuthology.orchestra.run.smithi123.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T06:21:57.203 INFO:teuthology.orchestra.run.smithi123.stdout:HOME_URL="https://centos.org/" 2023-11-15T06:21:57.203 INFO:teuthology.orchestra.run.smithi123.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T06:21:57.203 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T06:21:57.204 INFO:teuthology.orchestra.run.smithi123.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T06:21:57.204 INFO:teuthology.lock.ops:Updating smithi123.front.sepia.ceph.com on lock server 2023-11-15T06:21:57.233 DEBUG:teuthology.orchestra.run.smithi135:> uname -m 2023-11-15T06:21:57.255 INFO:teuthology.orchestra.run.smithi135.stdout:x86_64 2023-11-15T06:21:57.256 DEBUG:teuthology.orchestra.run.smithi135:> cat /etc/os-release 2023-11-15T06:21:57.317 INFO:teuthology.orchestra.run.smithi135.stdout:NAME="CentOS Stream" 2023-11-15T06:21:57.317 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION="8" 2023-11-15T06:21:57.317 INFO:teuthology.orchestra.run.smithi135.stdout:ID="centos" 2023-11-15T06:21:57.317 INFO:teuthology.orchestra.run.smithi135.stdout:ID_LIKE="rhel fedora" 2023-11-15T06:21:57.318 INFO:teuthology.orchestra.run.smithi135.stdout:VERSION_ID="8" 2023-11-15T06:21:57.318 INFO:teuthology.orchestra.run.smithi135.stdout:PLATFORM_ID="platform:el8" 2023-11-15T06:21:57.318 INFO:teuthology.orchestra.run.smithi135.stdout:PRETTY_NAME="CentOS Stream 8" 2023-11-15T06:21:57.318 INFO:teuthology.orchestra.run.smithi135.stdout:ANSI_COLOR="0;31" 2023-11-15T06:21:57.319 INFO:teuthology.orchestra.run.smithi135.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2023-11-15T06:21:57.319 INFO:teuthology.orchestra.run.smithi135.stdout:HOME_URL="https://centos.org/" 2023-11-15T06:21:57.319 INFO:teuthology.orchestra.run.smithi135.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2023-11-15T06:21:57.319 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2023-11-15T06:21:57.320 INFO:teuthology.orchestra.run.smithi135.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2023-11-15T06:21:57.320 INFO:teuthology.lock.ops:Updating smithi135.front.sepia.ceph.com on lock server 2023-11-15T06:21:57.345 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2023-11-15T06:21:57.355 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2023-11-15T06:21:57.360 INFO:teuthology.task.internal:Checking for old test directory... 2023-11-15T06:21:57.361 DEBUG:teuthology.orchestra.run.smithi123:> test '!' -e /home/ubuntu/cephtest 2023-11-15T06:21:57.365 DEBUG:teuthology.orchestra.run.smithi135:> test '!' -e /home/ubuntu/cephtest 2023-11-15T06:21:57.385 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2023-11-15T06:21:57.394 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2023-11-15T06:21:57.395 DEBUG:teuthology.orchestra.run.smithi123:> test -z $(ls -A /var/lib/ceph) 2023-11-15T06:21:57.426 DEBUG:teuthology.orchestra.run.smithi135:> test -z $(ls -A /var/lib/ceph) 2023-11-15T06:21:57.457 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2023-11-15T06:21:57.508 INFO:teuthology.run_tasks:Running task kernel... 2023-11-15T06:21:57.524 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2023-11-15T06:21:57.524 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2023-11-15T06:21:57.525 DEBUG:teuthology.orchestra.run.smithi123:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T06:21:57.526 DEBUG:teuthology.orchestra.run.smithi135:> test -f /run/.containerenv -o -f /.dockerenv 2023-11-15T06:21:57.547 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:21:57.547 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2023-11-15T06:21:57.548 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:21:57.549 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-11-15T06:21:57.608 INFO:teuthology.orchestra.run.smithi135.stdout:4.18.0-489.el8.x86_64 2023-11-15T06:21:57.609 INFO:teuthology.task.kernel:Running kernel on smithi135: 4.18.0-489.el8.x86_64 2023-11-15T06:21:57.609 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum install -y kernel 2023-11-15T06:21:57.610 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-489.el8.x86_64 2023-11-15T06:21:57.611 INFO:teuthology.task.kernel:Running kernel on smithi123: 4.18.0-489.el8.x86_64 2023-11-15T06:21:57.611 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum install -y kernel 2023-11-15T06:22:02.908 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-15T06:22:03.085 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-15T06:22:12.577 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - BaseOS 16 MB/s | 53 MB 00:03 2023-11-15T06:22:14.513 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 10 MB/s | 53 MB 00:05 2023-11-15T06:22:20.252 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras 159 kB/s | 18 kB 00:00 2023-11-15T06:22:20.803 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras common packages 19 kB/s | 6.9 kB 00:00 2023-11-15T06:22:21.096 INFO:teuthology.orchestra.run.smithi135.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2023-11-15T06:22:22.207 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 148 kB/s | 18 kB 00:00 2023-11-15T06:22:22.774 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2023-11-15T06:22:22.956 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 9.3 MB/s | 16 MB 00:01 2023-11-15T06:22:23.059 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 26 kB/s | 3.5 kB 00:00 2023-11-15T06:22:24.220 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 16 MB/s | 16 MB 00:01 2023-11-15T06:22:26.772 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 412 kB/s | 24 kB 00:00 2023-11-15T06:22:27.993 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 419 kB/s | 24 kB 00:00 2023-11-15T06:22:30.389 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T06:22:30.389 INFO:teuthology.orchestra.run.smithi135.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T06:22:30.677 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:22:30.678 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:22:30.678 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:22:30.679 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:22:30.679 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:22:30.679 INFO:teuthology.orchestra.run.smithi135.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T06:22:30.679 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:22:30.679 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T06:22:30.680 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T06:22:30.680 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:22:30.680 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:22:30.680 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:22:30.680 INFO:teuthology.orchestra.run.smithi135.stdout:Install 3 Packages 2023-11-15T06:22:30.680 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:22:30.681 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 89 M 2023-11-15T06:22:30.681 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 96 M 2023-11-15T06:22:30.682 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:22:31.526 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2023-11-15T06:22:31.526 INFO:teuthology.orchestra.run.smithi123.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2023-11-15T06:22:31.814 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:22:31.815 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:22:31.815 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:22:31.816 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:22:31.816 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:22:31.816 INFO:teuthology.orchestra.run.smithi123.stdout: kernel x86_64 4.18.0-521.el8 baseos 10 M 2023-11-15T06:22:31.816 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:22:31.816 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-core x86_64 4.18.0-521.el8 baseos 43 M 2023-11-15T06:22:31.817 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules x86_64 4.18.0-521.el8 baseos 36 M 2023-11-15T06:22:31.817 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:22:31.817 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:22:31.817 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:22:31.817 INFO:teuthology.orchestra.run.smithi123.stdout:Install 3 Packages 2023-11-15T06:22:31.818 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:22:31.818 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 89 M 2023-11-15T06:22:31.819 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 96 M 2023-11-15T06:22:31.819 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:22:32.033 INFO:teuthology.orchestra.run.smithi135.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 8.6 MB/s | 10 MB 00:01 2023-11-15T06:22:33.258 INFO:teuthology.orchestra.run.smithi135.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2023-11-15T06:22:33.311 INFO:teuthology.orchestra.run.smithi123.stdout:(1/3): kernel-4.18.0-521.el8.x86_64.rpm 7.7 MB/s | 10 MB 00:01 2023-11-15T06:22:33.850 INFO:teuthology.orchestra.run.smithi135.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 14 MB/s | 43 MB 00:02 2023-11-15T06:22:33.850 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:33.851 INFO:teuthology.orchestra.run.smithi135.stdout:Total 28 MB/s | 89 MB 00:03 2023-11-15T06:22:34.411 INFO:teuthology.orchestra.run.smithi123.stdout:(2/3): kernel-modules-4.18.0-521.el8.x86_64.rpm 15 MB/s | 36 MB 00:02 2023-11-15T06:22:34.525 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:22:34.940 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:22:34.940 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:22:34.944 INFO:teuthology.orchestra.run.smithi123.stdout:(3/3): kernel-core-4.18.0-521.el8.x86_64.rpm 15 MB/s | 43 MB 00:02 2023-11-15T06:22:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:22:34.945 INFO:teuthology.orchestra.run.smithi123.stdout:Total 29 MB/s | 89 MB 00:03 2023-11-15T06:22:35.619 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:22:36.041 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:22:36.041 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:22:36.172 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:22:36.200 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:22:37.274 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:22:37.297 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:22:38.956 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:22:39.482 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:22:40.012 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:22:40.629 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:22:43.057 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:22:43.449 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:22:44.297 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:22:44.683 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:22:46.901 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:22:47.162 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:22:48.117 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:22:48.581 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:09.006 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:09.030 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:09.395 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:09.396 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:23:09.396 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:23:10.184 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:10.184 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:23:10.185 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:23:10.185 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T06:23:10.185 INFO:teuthology.orchestra.run.smithi135.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T06:23:10.185 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:23:10.185 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:23:10.319 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:23:10.387 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-core-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:10.410 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:10.541 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:23:10.541 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:23:10.541 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T06:23:10.542 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T06:23:10.542 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T06:23:10.542 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2023-11-15T06:23:10.542 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi135.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T06:23:10.543 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:23:10.746 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: kernel-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:10.747 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-4.18.0-521.el8.x86_64 1/3 2023-11-15T06:23:10.747 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-core-4.18.0-521.el8.x86_64 2/3 2023-11-15T06:23:10.751 INFO:teuthology.orchestra.run.smithi135.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:23:10.752 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:23:10.752 DEBUG:teuthology.orchestra.run.smithi135:> sudo rpm -qi grub2-tools 2023-11-15T06:23:10.786 INFO:teuthology.orchestra.run.smithi135.stdout:Name : grub2-tools 2023-11-15T06:23:10.787 INFO:teuthology.orchestra.run.smithi135.stdout:Epoch : 1 2023-11-15T06:23:10.787 INFO:teuthology.orchestra.run.smithi135.stdout:Version : 2.02 2023-11-15T06:23:10.787 INFO:teuthology.orchestra.run.smithi135.stdout:Release : 129.el8 2023-11-15T06:23:10.787 INFO:teuthology.orchestra.run.smithi135.stdout:Architecture: x86_64 2023-11-15T06:23:10.787 INFO:teuthology.orchestra.run.smithi135.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T06:23:10.788 INFO:teuthology.orchestra.run.smithi135.stdout:Group : System Environment/Base 2023-11-15T06:23:10.788 INFO:teuthology.orchestra.run.smithi135.stdout:Size : 9265586 2023-11-15T06:23:10.788 INFO:teuthology.orchestra.run.smithi135.stdout:License : GPLv3+ 2023-11-15T06:23:10.788 INFO:teuthology.orchestra.run.smithi135.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T06:23:10.788 INFO:teuthology.orchestra.run.smithi135.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T06:23:10.789 INFO:teuthology.orchestra.run.smithi135.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T06:23:10.789 INFO:teuthology.orchestra.run.smithi135.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T06:23:10.789 INFO:teuthology.orchestra.run.smithi135.stdout:Relocations : (not relocatable) 2023-11-15T06:23:10.789 INFO:teuthology.orchestra.run.smithi135.stdout:Packager : CentOS BuildSystem 2023-11-15T06:23:10.789 INFO:teuthology.orchestra.run.smithi135.stdout:Vendor : CentOS 2023-11-15T06:23:10.790 INFO:teuthology.orchestra.run.smithi135.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T06:23:10.790 INFO:teuthology.orchestra.run.smithi135.stdout:Summary : Support tools for GRUB. 2023-11-15T06:23:10.790 INFO:teuthology.orchestra.run.smithi135.stdout:Description : 2023-11-15T06:23:10.790 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:23:10.790 INFO:teuthology.orchestra.run.smithi135.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T06:23:10.791 INFO:teuthology.orchestra.run.smithi135.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T06:23:10.791 INFO:teuthology.orchestra.run.smithi135.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T06:23:10.791 INFO:teuthology.orchestra.run.smithi135.stdout:hardware devices. 2023-11-15T06:23:10.791 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:23:10.791 INFO:teuthology.orchestra.run.smithi135.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T06:23:10.792 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T06:23:10.793 INFO:teuthology.task.kernel:Updating grub on smithi135 to boot 4.18.0-521.el8.x86_64 2023-11-15T06:23:10.793 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T06:23:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : kernel-modules-4.18.0-521.el8.x86_64 3/3 2023-11-15T06:23:11.501 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:23:11.501 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:23:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-4.18.0-521.el8.x86_64 kernel-core-4.18.0-521.el8.x86_64 2023-11-15T06:23:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: kernel-modules-4.18.0-521.el8.x86_64 2023-11-15T06:23:11.502 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:23:11.502 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:23:11.642 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:23:11.859 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:23:11.860 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:23:11.860 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-521.el8.x86_64 2023-11-15T06:23:11.860 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2023-11-15T06:23:11.861 DEBUG:teuthology.task.kernel:src is distro, skipping download 2023-11-15T06:23:11.861 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2023-11-15T06:23:11.861 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi123.front.sepia.ceph.com, path=None, version=distro) 2023-11-15T06:23:11.861 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q kernel | sort -rV | head -n 1 2023-11-15T06:23:12.069 INFO:teuthology.orchestra.run.smithi123.stdout:kernel-4.18.0-521.el8.x86_64 2023-11-15T06:23:12.069 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-521.el8.x86_64 2023-11-15T06:23:12.070 DEBUG:teuthology.orchestra.run.smithi123:> sudo rpm -qi grub2-tools 2023-11-15T06:23:12.104 INFO:teuthology.orchestra.run.smithi123.stdout:Name : grub2-tools 2023-11-15T06:23:12.104 INFO:teuthology.orchestra.run.smithi123.stdout:Epoch : 1 2023-11-15T06:23:12.104 INFO:teuthology.orchestra.run.smithi123.stdout:Version : 2.02 2023-11-15T06:23:12.104 INFO:teuthology.orchestra.run.smithi123.stdout:Release : 129.el8 2023-11-15T06:23:12.105 INFO:teuthology.orchestra.run.smithi123.stdout:Architecture: x86_64 2023-11-15T06:23:12.105 INFO:teuthology.orchestra.run.smithi123.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2023-11-15T06:23:12.105 INFO:teuthology.orchestra.run.smithi123.stdout:Group : System Environment/Base 2023-11-15T06:23:12.105 INFO:teuthology.orchestra.run.smithi123.stdout:Size : 9265586 2023-11-15T06:23:12.105 INFO:teuthology.orchestra.run.smithi123.stdout:License : GPLv3+ 2023-11-15T06:23:12.106 INFO:teuthology.orchestra.run.smithi123.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2023-11-15T06:23:12.106 INFO:teuthology.orchestra.run.smithi123.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2023-11-15T06:23:12.106 INFO:teuthology.orchestra.run.smithi123.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2023-11-15T06:23:12.106 INFO:teuthology.orchestra.run.smithi123.stdout:Build Host : kbuilder.bsys.centos.org 2023-11-15T06:23:12.106 INFO:teuthology.orchestra.run.smithi123.stdout:Relocations : (not relocatable) 2023-11-15T06:23:12.107 INFO:teuthology.orchestra.run.smithi123.stdout:Packager : CentOS BuildSystem 2023-11-15T06:23:12.107 INFO:teuthology.orchestra.run.smithi123.stdout:Vendor : CentOS 2023-11-15T06:23:12.107 INFO:teuthology.orchestra.run.smithi123.stdout:URL : http://www.gnu.org/software/grub/ 2023-11-15T06:23:12.107 INFO:teuthology.orchestra.run.smithi123.stdout:Summary : Support tools for GRUB. 2023-11-15T06:23:12.108 INFO:teuthology.orchestra.run.smithi123.stdout:Description : 2023-11-15T06:23:12.108 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:23:12.108 INFO:teuthology.orchestra.run.smithi123.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2023-11-15T06:23:12.108 INFO:teuthology.orchestra.run.smithi123.stdout:customizable bootloader with modular architecture. It supports a rich 2023-11-15T06:23:12.108 INFO:teuthology.orchestra.run.smithi123.stdout:variety of kernel formats, file systems, computer architectures and 2023-11-15T06:23:12.109 INFO:teuthology.orchestra.run.smithi123.stdout:hardware devices. 2023-11-15T06:23:12.109 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:23:12.109 INFO:teuthology.orchestra.run.smithi123.stdout:This subpackage provides tools for support of all platforms. 2023-11-15T06:23:12.110 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2023-11-15T06:23:12.110 INFO:teuthology.task.kernel:Updating grub on smithi123 to boot 4.18.0-521.el8.x86_64 2023-11-15T06:23:12.111 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2023-11-15T06:23:12.706 INFO:teuthology.orchestra.run.smithi135.stderr:Generating grub configuration file ... 2023-11-15T06:23:13.731 INFO:teuthology.orchestra.run.smithi123.stderr:Generating grub configuration file ... 2023-11-15T06:23:14.878 INFO:teuthology.orchestra.run.smithi135.stderr:done 2023-11-15T06:23:14.880 DEBUG:teuthology.orchestra.run.smithi135:> mktemp 2023-11-15T06:23:14.916 INFO:teuthology.orchestra.run.smithi135.stdout:/tmp/tmp.mYz3UTj6VO 2023-11-15T06:23:14.916 DEBUG:teuthology.orchestra.run.smithi135:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.mYz3UTj6VO 2023-11-15T06:23:14.995 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 0666 /tmp/tmp.mYz3UTj6VO 2023-11-15T06:23:15.143 DEBUG:teuthology.orchestra.remote:smithi135:/tmp/tmp.mYz3UTj6VO is 6KB 2023-11-15T06:23:15.191 DEBUG:teuthology.orchestra.run.smithi135:> rm -fr /tmp/tmp.mYz3UTj6VO 2023-11-15T06:23:15.209 DEBUG:teuthology.orchestra.run.smithi135:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T06:23:15.275 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T06:23:15.275 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T06:23:15.275 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T06:23:15.276 INFO:teuthology.orchestra.run.smithi135.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T06:23:15.277 DEBUG:teuthology.orchestra.run.smithi135:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T06:23:15.417 DEBUG:teuthology.orchestra.run.smithi135:> sudo shutdown -r now 2023-11-15T06:23:15.500 INFO:teuthology.orchestra.run.smithi123.stderr:done 2023-11-15T06:23:15.502 DEBUG:teuthology.orchestra.run.smithi123:> mktemp 2023-11-15T06:23:15.526 INFO:teuthology.orchestra.run.smithi123.stdout:/tmp/tmp.FM2X9nKe35 2023-11-15T06:23:15.526 DEBUG:teuthology.orchestra.run.smithi123:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.FM2X9nKe35 2023-11-15T06:23:15.606 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 0666 /tmp/tmp.FM2X9nKe35 2023-11-15T06:23:15.750 DEBUG:teuthology.orchestra.remote:smithi123:/tmp/tmp.FM2X9nKe35 is 6KB 2023-11-15T06:23:15.799 DEBUG:teuthology.orchestra.run.smithi123:> rm -fr /tmp/tmp.FM2X9nKe35 2023-11-15T06:23:15.817 DEBUG:teuthology.orchestra.run.smithi123:> sudo /bin/ls /boot/loader/entries || true 2023-11-15T06:23:15.890 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2023-11-15T06:23:15.891 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2023-11-15T06:23:15.891 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2023-11-15T06:23:15.891 INFO:teuthology.orchestra.run.smithi123.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64.conf 2023-11-15T06:23:15.892 DEBUG:teuthology.orchestra.run.smithi123:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-521.el8.x86_64 2023-11-15T06:23:16.053 DEBUG:teuthology.orchestra.run.smithi123:> sudo shutdown -r now 2023-11-15T06:23:45.447 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2023-11-15T06:23:45.447 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-11-15T06:23:45.448 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:23:46.056 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2023-11-15T06:23:46.057 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-15T06:23:46.057 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:24:03.962 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-11-15T06:24:04.478 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-11-15T06:24:12.970 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-11-15T06:24:12.971 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:24:13.480 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-15T06:24:13.480 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:24:16.026 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.135 2023-11-15T06:24:16.538 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.123 2023-11-15T06:24:28.038 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-11-15T06:24:28.039 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:24:28.540 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-15T06:24:28.541 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:25:00.257 DEBUG:teuthology.orchestra.run.smithi135:> true 2023-11-15T06:25:00.714 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-11-15T06:25:00.973 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-15T06:25:00.973 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-521.el8.x86_64"... 2023-11-15T06:25:00.974 DEBUG:teuthology.orchestra.run.smithi123:> uname -r 2023-11-15T06:25:01.034 INFO:teuthology.orchestra.run.smithi123.stdout:4.18.0-521.el8.x86_64 2023-11-15T06:25:01.034 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T06:25:01.035 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T06:25:01.035 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T06:25:01.349 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-11-15T06:25:01.349 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-521.el8.x86_64"... 2023-11-15T06:25:01.350 DEBUG:teuthology.orchestra.run.smithi135:> uname -r 2023-11-15T06:25:01.800 INFO:teuthology.orchestra.run.smithi135.stdout:4.18.0-521.el8.x86_64 2023-11-15T06:25:01.801 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-521.el8.x86_64 vs 4.18.0-521.el8.x86_64 2023-11-15T06:25:01.801 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2023-11-15T06:25:01.801 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2023-11-15T06:25:02.036 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2023-11-15T06:25:02.036 DEBUG:teuthology.orchestra.run.smithi123:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T06:25:02.125 INFO:teuthology.orchestra.run.smithi123.stdout:ttyS1 2023-11-15T06:25:02.139 DEBUG:teuthology.parallel:result is None 2023-11-15T06:25:02.803 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2023-11-15T06:25:02.803 DEBUG:teuthology.orchestra.run.smithi135:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2023-11-15T06:25:02.886 INFO:teuthology.orchestra.run.smithi135.stdout:ttyS1 2023-11-15T06:25:02.897 DEBUG:teuthology.parallel:result is None 2023-11-15T06:25:02.898 INFO:teuthology.run_tasks:Running task internal.base... 2023-11-15T06:25:02.906 INFO:teuthology.task.internal:Creating test directory... 2023-11-15T06:25:02.907 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T06:25:02.910 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2023-11-15T06:25:02.929 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2023-11-15T06:25:02.936 INFO:teuthology.run_tasks:Running task internal.archive... 2023-11-15T06:25:02.942 INFO:teuthology.task.internal:Creating archive directory... 2023-11-15T06:25:02.943 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T06:25:02.972 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2023-11-15T06:25:03.017 INFO:teuthology.run_tasks:Running task internal.coredump... 2023-11-15T06:25:03.024 INFO:teuthology.task.internal:Enabling coredump saving... 2023-11-15T06:25:03.024 DEBUG:teuthology.orchestra.run.smithi123:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T06:25:03.054 DEBUG:teuthology.orchestra.run.smithi135:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2023-11-15T06:25:03.090 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:25:03.098 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:25:03.102 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:25:03.107 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2023-11-15T06:25:03.109 INFO:teuthology.run_tasks:Running task internal.sudo... 2023-11-15T06:25:03.116 INFO:teuthology.task.internal:Configuring sudo... 2023-11-15T06:25:03.117 DEBUG:teuthology.orchestra.run.smithi123:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T06:25:03.146 DEBUG:teuthology.orchestra.run.smithi135:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2023-11-15T06:25:03.178 INFO:teuthology.run_tasks:Running task internal.syslog... 2023-11-15T06:25:03.187 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2023-11-15T06:25:03.188 DEBUG:teuthology.orchestra.run.smithi123:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T06:25:03.221 DEBUG:teuthology.orchestra.run.smithi135:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2023-11-15T06:25:03.240 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:25:03.328 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:25:03.393 DEBUG:teuthology.orchestra.run.smithi123:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:25:03.443 DEBUG:teuthology.orchestra.run.smithi123:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:25:03.490 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:25:03.490 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T06:25:03.563 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:25:03.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2023-11-15T06:25:03.671 DEBUG:teuthology.orchestra.run.smithi135:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:25:03.714 DEBUG:teuthology.orchestra.run.smithi135:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2023-11-15T06:25:03.756 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:25:03.757 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2023-11-15T06:25:03.822 DEBUG:teuthology.orchestra.run.smithi123:> sudo service rsyslog restart 2023-11-15T06:25:03.825 DEBUG:teuthology.orchestra.run.smithi135:> sudo service rsyslog restart 2023-11-15T06:25:03.886 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:25:03.924 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:25:04.168 INFO:teuthology.run_tasks:Running task internal.timer... 2023-11-15T06:25:04.176 INFO:teuthology.task.internal:Starting timer... 2023-11-15T06:25:04.177 INFO:teuthology.run_tasks:Running task pcp... 2023-11-15T06:25:04.261 INFO:teuthology.run_tasks:Running task selinux... 2023-11-15T06:25:04.270 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2023-11-15T06:25:04.271 DEBUG:teuthology.orchestra.run.smithi123:> sudo service auditd rotate 2023-11-15T06:25:04.367 INFO:teuthology.orchestra.run.smithi123.stdout:Rotating logs: 2023-11-15T06:25:04.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo service auditd rotate 2023-11-15T06:25:04.472 INFO:teuthology.orchestra.run.smithi135.stdout:Rotating logs: 2023-11-15T06:25:04.474 DEBUG:teuthology.task.selinux:Getting current SELinux state 2023-11-15T06:25:04.474 DEBUG:teuthology.orchestra.run.smithi123:> /usr/sbin/getenforce 2023-11-15T06:25:04.500 INFO:teuthology.orchestra.run.smithi123.stdout:Permissive 2023-11-15T06:25:04.500 DEBUG:teuthology.orchestra.run.smithi135:> /usr/sbin/getenforce 2023-11-15T06:25:04.534 INFO:teuthology.orchestra.run.smithi135.stdout:Permissive 2023-11-15T06:25:04.534 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi123.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi135.front.sepia.ceph.com': 'permissive'} 2023-11-15T06:25:04.535 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:25:04.568 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:25:04.569 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:25:04.599 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:25:04.600 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2023-11-15T06:25:04.600 DEBUG:teuthology.orchestra.run.smithi123:> sudo /usr/sbin/setenforce permissive 2023-11-15T06:25:04.636 DEBUG:teuthology.orchestra.run.smithi135:> sudo /usr/sbin/setenforce permissive 2023-11-15T06:25:04.667 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2023-11-15T06:25:04.678 DEBUG:teuthology.repo_utils:Setting repo remote to https://git.ceph.com/ceph-cm-ansible.git 2023-11-15T06:25:04.684 INFO:teuthology.repo_utils:Fetching main from origin 2023-11-15T06:25:04.809 INFO:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2023-11-15T06:25:04.828 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2023-11-15T06:25:04.830 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi123.front.sepia.ceph.com,smithi135.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2023-11-15T06:32:37.578 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi123.front.sepia.ceph.com'), Remote(name='ubuntu@smithi135.front.sepia.ceph.com')] 2023-11-15T06:32:37.580 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-15T06:32:37.581 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi123.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:32:37.650 DEBUG:teuthology.orchestra.run.smithi123:> true 2023-11-15T06:32:37.716 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi123.front.sepia.ceph.com' 2023-11-15T06:32:37.716 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-11-15T06:32:37.717 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi135.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2023-11-15T06:32:37.787 DEBUG:teuthology.orchestra.run.smithi135:> true 2023-11-15T06:32:37.863 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi135.front.sepia.ceph.com' 2023-11-15T06:32:37.864 INFO:teuthology.run_tasks:Running task clock... 2023-11-15T06:32:37.876 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2023-11-15T06:32:37.877 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T06:32:37.878 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:32:37.880 INFO:teuthology.orchestra.run:Running command with timeout 360 2023-11-15T06:32:37.881 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:32:37.909 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T06:32:37.922 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T06:32:37.939 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2023-11-15T06:32:37.953 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2023-11-15T06:32:37.994 INFO:teuthology.orchestra.run.smithi123.stderr:sudo: ntpd: command not found 2023-11-15T06:32:38.005 INFO:teuthology.orchestra.run.smithi123.stdout:506 Cannot talk to daemon 2023-11-15T06:32:38.019 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T06:32:38.034 INFO:teuthology.orchestra.run.smithi135.stderr:sudo: ntpd: command not found 2023-11-15T06:32:38.035 INFO:teuthology.orchestra.run.smithi123.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T06:32:38.044 INFO:teuthology.orchestra.run.smithi135.stdout:506 Cannot talk to daemon 2023-11-15T06:32:38.058 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntp.service: Unit ntp.service not found. 2023-11-15T06:32:38.071 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2023-11-15T06:32:38.102 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-11-15T06:32:38.106 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:32:38.106 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-11-15T06:32:38.106 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.107 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.107 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.107 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.145 INFO:teuthology.orchestra.run.smithi135.stderr:bash: ntpq: command not found 2023-11-15T06:32:38.148 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:32:38.149 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2023-11-15T06:32:38.149 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.149 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.149 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.149 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:32:38.150 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T06:32:38.160 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T06:32:38.161 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi123.front.sepia.ceph.com 2023-11-15T06:32:38.161 DEBUG:teuthology.orchestra.run.smithi123:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:32:38.162 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi135.front.sepia.ceph.com 2023-11-15T06:32:38.162 DEBUG:teuthology.orchestra.run.smithi135:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:32:39.039 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 Nov 2023 06:31:02 AM UTC. 2023-11-15T06:32:39.065 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:36 ago on Wed 15 Nov 2023 06:31:02 AM UTC. 2023-11-15T06:32:41.112 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:32:41.114 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:32:41.114 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:32:41.115 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:32:41.115 INFO:teuthology.orchestra.run.smithi135.stdout:Resetting modules: 2023-11-15T06:32:41.115 INFO:teuthology.orchestra.run.smithi135.stdout: container-tools 2023-11-15T06:32:41.115 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:32:41.115 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:32:41.116 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:32:41.116 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:32:41.197 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:32:41.198 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:32:41.198 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:32:41.198 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:32:41.198 INFO:teuthology.orchestra.run.smithi123.stdout:Resetting modules: 2023-11-15T06:32:41.199 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools 2023-11-15T06:32:41.199 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:32:41.199 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:32:41.199 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:32:41.200 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:32:41.359 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:32:41.459 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:32:42.305 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 Nov 2023 06:31:02 AM UTC. 2023-11-15T06:32:42.406 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:40 ago on Wed 15 Nov 2023 06:31:02 AM UTC. 2023-11-15T06:32:44.413 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:32:44.496 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:32:44.592 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:32:44.592 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-11-15T06:32:44.592 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:32:44.593 INFO:teuthology.orchestra.run.smithi135.stdout:Installing group/module packages: 2023-11-15T06:32:44.593 INFO:teuthology.orchestra.run.smithi135.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T06:32:44.594 INFO:teuthology.orchestra.run.smithi135.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T06:32:44.594 INFO:teuthology.orchestra.run.smithi135.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T06:32:44.594 INFO:teuthology.orchestra.run.smithi135.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T06:32:44.595 INFO:teuthology.orchestra.run.smithi135.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T06:32:44.595 INFO:teuthology.orchestra.run.smithi135.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T06:32:44.595 INFO:teuthology.orchestra.run.smithi135.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T06:32:44.596 INFO:teuthology.orchestra.run.smithi135.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T06:32:44.596 INFO:teuthology.orchestra.run.smithi135.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T06:32:44.597 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:32:44.597 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T06:32:44.597 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T06:32:44.598 INFO:teuthology.orchestra.run.smithi135.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T06:32:44.598 INFO:teuthology.orchestra.run.smithi135.stdout:Installing weak dependencies: 2023-11-15T06:32:44.598 INFO:teuthology.orchestra.run.smithi135.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T06:32:44.599 INFO:teuthology.orchestra.run.smithi135.stdout:Installing module profiles: 2023-11-15T06:32:44.599 INFO:teuthology.orchestra.run.smithi135.stdout: container-tools/common 2023-11-15T06:32:44.599 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:32:44.599 INFO:teuthology.orchestra.run.smithi135.stdout:Enabling module streams: 2023-11-15T06:32:44.600 INFO:teuthology.orchestra.run.smithi135.stdout: container-tools rhel8 2023-11-15T06:32:44.600 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:32:44.600 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:32:44.600 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:32:44.601 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2023-11-15T06:32:44.601 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:32:44.602 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 27 M 2023-11-15T06:32:44.603 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 89 M 2023-11-15T06:32:44.603 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:32:44.674 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:32:44.674 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-15T06:32:44.674 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:32:44.675 INFO:teuthology.orchestra.run.smithi123.stdout:Installing group/module packages: 2023-11-15T06:32:44.675 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2023-11-15T06:32:44.675 INFO:teuthology.orchestra.run.smithi123.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2023-11-15T06:32:44.675 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2023-11-15T06:32:44.675 INFO:teuthology.orchestra.run.smithi123.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2023-11-15T06:32:44.676 INFO:teuthology.orchestra.run.smithi123.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2023-11-15T06:32:44.676 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2023-11-15T06:32:44.676 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2023-11-15T06:32:44.676 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2023-11-15T06:32:44.676 INFO:teuthology.orchestra.run.smithi123.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2023-11-15T06:32:44.677 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:32:44.677 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2023-11-15T06:32:44.677 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2023-11-15T06:32:44.677 INFO:teuthology.orchestra.run.smithi123.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2023-11-15T06:32:44.677 INFO:teuthology.orchestra.run.smithi123.stdout:Installing weak dependencies: 2023-11-15T06:32:44.678 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2023-11-15T06:32:44.678 INFO:teuthology.orchestra.run.smithi123.stdout:Installing module profiles: 2023-11-15T06:32:44.678 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools/common 2023-11-15T06:32:44.678 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:32:44.678 INFO:teuthology.orchestra.run.smithi123.stdout:Enabling module streams: 2023-11-15T06:32:44.679 INFO:teuthology.orchestra.run.smithi123.stdout: container-tools rhel8 2023-11-15T06:32:44.679 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:32:44.679 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:32:44.679 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:32:44.679 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2023-11-15T06:32:44.680 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:32:44.680 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 27 M 2023-11-15T06:32:44.681 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 89 M 2023-11-15T06:32:44.681 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:32:44.921 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.5 MB/s | 750 kB 00:00 2023-11-15T06:32:44.953 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.1 MB/s | 1.0 MB 00:00 2023-11-15T06:32:45.004 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 453 kB/s | 38 kB 00:00 2023-11-15T06:32:45.037 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.8 MB/s | 241 kB 00:00 2023-11-15T06:32:45.096 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2023-11-15T06:32:45.144 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 2.7 MB/s | 750 kB 00:00 2023-11-15T06:32:45.146 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 495 kB/s | 25 kB 00:00 2023-11-15T06:32:45.178 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 3.2 MB/s | 1.0 MB 00:00 2023-11-15T06:32:45.203 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 649 kB/s | 38 kB 00:00 2023-11-15T06:32:45.213 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.4 MB/s | 94 kB 00:00 2023-11-15T06:32:45.270 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.6 MB/s | 241 kB 00:00 2023-11-15T06:32:45.330 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 3.0 MB/s | 178 kB 00:00 2023-11-15T06:32:45.387 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 424 kB/s | 25 kB 00:00 2023-11-15T06:32:45.430 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.2 MB/s | 3.9 MB 00:00 2023-11-15T06:32:45.446 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.6 MB/s | 94 kB 00:00 2023-11-15T06:32:45.563 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 11 MB/s | 3.9 MB 00:00 2023-11-15T06:32:45.747 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 8.6 MB/s | 9.6 MB 00:01 2023-11-15T06:32:45.813 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): toolbox-0.0.99.4-6.module_el8+712+4cd1b 10 MB/s | 2.5 MB 00:00 2023-11-15T06:32:45.814 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 6.6 MB/s | 2.5 MB 00:00 2023-11-15T06:32:45.840 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 572 kB/s | 53 kB 00:00 2023-11-15T06:32:45.873 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 699 kB/s | 41 kB 00:00 2023-11-15T06:32:45.955 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): buildah-1.32.0-1.module_el8+712+4cd1bd 8.8 MB/s | 9.6 MB 00:01 2023-11-15T06:32:45.981 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): udica-0.2.6-20.module_el8+712+4cd1bd69 315 kB/s | 53 kB 00:00 2023-11-15T06:32:46.006 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 817 kB/s | 41 kB 00:00 2023-11-15T06:32:46.103 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 9.2 MB/s | 8.1 MB 00:00 2023-11-15T06:32:46.103 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:32:46.103 INFO:teuthology.orchestra.run.smithi135.stdout:Total 18 MB/s | 27 MB 00:01 2023-11-15T06:32:46.366 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:32:46.387 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:32:46.388 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:32:46.448 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 8.1 MB/s | 8.1 MB 00:01 2023-11-15T06:32:46.448 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:32:46.449 INFO:teuthology.orchestra.run.smithi123.stdout:Total 15 MB/s | 27 MB 00:01 2023-11-15T06:32:46.646 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:32:46.650 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:32:46.716 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:32:46.737 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:32:46.737 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:32:46.990 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:32:46.993 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:32:47.351 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:32:47.621 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T06:32:47.727 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:32:47.800 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T06:32:47.969 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2023-11-15T06:32:47.998 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T06:32:48.114 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2023-11-15T06:32:48.245 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:32:48.324 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2023-11-15T06:32:48.541 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:32:48.831 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T06:32:49.075 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2023-11-15T06:32:49.834 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:32:50.047 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:32:50.059 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T06:32:50.248 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T06:32:50.295 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2023-11-15T06:32:50.494 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2023-11-15T06:32:50.554 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T06:32:50.735 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2023-11-15T06:32:50.946 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T06:32:51.126 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2023-11-15T06:32:51.857 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:32:52.031 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:32:52.091 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T06:32:52.235 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2023-11-15T06:32:52.283 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:32:52.421 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:32:52.703 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:32:52.703 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T06:32:52.704 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T06:32:52.704 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T06:32:52.704 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:32:52.704 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T06:32:52.704 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:32:52.705 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T06:32:52.705 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T06:32:52.705 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T06:32:52.705 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T06:32:52.705 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:32:52.706 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T06:32:52.838 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2023-11-15T06:32:52.838 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2023-11-15T06:32:52.839 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2023-11-15T06:32:52.839 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2023-11-15T06:32:52.839 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2023-11-15T06:32:52.840 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2023-11-15T06:32:52.840 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2023-11-15T06:32:52.840 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2023-11-15T06:32:52.841 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2023-11-15T06:32:52.841 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2023-11-15T06:32:52.842 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2023-11-15T06:32:52.842 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2023-11-15T06:32:52.842 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2023-11-15T06:32:53.510 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T06:32:53.511 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:32:53.511 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:32:53.511 INFO:teuthology.orchestra.run.smithi135.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.511 INFO:teuthology.orchestra.run.smithi135.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.511 INFO:teuthology.orchestra.run.smithi135.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:32:53.512 INFO:teuthology.orchestra.run.smithi135.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T06:32:53.512 INFO:teuthology.orchestra.run.smithi135.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.512 INFO:teuthology.orchestra.run.smithi135.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.512 INFO:teuthology.orchestra.run.smithi135.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:32:53.512 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T06:32:53.513 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T06:32:53.513 INFO:teuthology.orchestra.run.smithi135.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.513 INFO:teuthology.orchestra.run.smithi135.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.513 INFO:teuthology.orchestra.run.smithi135.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T06:32:53.513 INFO:teuthology.orchestra.run.smithi135.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T06:32:53.514 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:32:53.514 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:32:53.768 DEBUG:teuthology.parallel:result is None 2023-11-15T06:32:53.799 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2023-11-15T06:32:53.799 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:32:53.799 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:32:53.800 INFO:teuthology.orchestra.run.smithi123.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.800 INFO:teuthology.orchestra.run.smithi123.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.801 INFO:teuthology.orchestra.run.smithi123.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:32:53.801 INFO:teuthology.orchestra.run.smithi123.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2023-11-15T06:32:53.801 INFO:teuthology.orchestra.run.smithi123.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.802 INFO:teuthology.orchestra.run.smithi123.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.802 INFO:teuthology.orchestra.run.smithi123.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2023-11-15T06:32:53.803 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2023-11-15T06:32:53.803 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyxdg-0.25-16.el8.noarch 2023-11-15T06:32:53.803 INFO:teuthology.orchestra.run.smithi123.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.804 INFO:teuthology.orchestra.run.smithi123.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2023-11-15T06:32:53.804 INFO:teuthology.orchestra.run.smithi123.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2023-11-15T06:32:53.804 INFO:teuthology.orchestra.run.smithi123.stdout: yajl-2.1.0-12.el8.x86_64 2023-11-15T06:32:53.805 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:32:53.806 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:32:54.042 DEBUG:teuthology.parallel:result is None 2023-11-15T06:32:54.042 INFO:teuthology.run_tasks:Running task pexec... 2023-11-15T06:32:54.055 INFO:teuthology.task.pexec:Executing custom commands... 2023-11-15T06:32:54.056 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi123.front.sepia.ceph.com 2023-11-15T06:32:54.056 DEBUG:teuthology.orchestra.run.smithi123:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:32:54.057 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi135.front.sepia.ceph.com 2023-11-15T06:32:54.057 DEBUG:teuthology.orchestra.run.smithi135:> TESTDIR=/home/ubuntu/cephtest bash -s 2023-11-15T06:32:54.244 DEBUG:teuthology.parallel:result is None 2023-11-15T06:32:54.258 DEBUG:teuthology.parallel:result is None 2023-11-15T06:32:54.261 INFO:teuthology.run_tasks:Running task install... 2023-11-15T06:32:54.270 DEBUG:teuthology.task.install:project ceph 2023-11-15T06:32:54.270 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'}} 2023-11-15T06:32:54.271 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T06:32:54.271 INFO:teuthology.task.install:Using flavor: default 2023-11-15T06:32:54.278 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2023-11-15T06:32:54.278 INFO:teuthology.task.install:extra packages: [] 2023-11-15T06:32:54.279 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T06:32:54.280 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:32:54.280 INFO:teuthology.packaging:ref: None 2023-11-15T06:32:54.280 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:32:54.281 INFO:teuthology.packaging:branch: None 2023-11-15T06:32:54.281 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:32:54.470 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:32:54.471 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:32:54.473 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae', 'tag': 'v16.2.4', 'wait_for_package': False} 2023-11-15T06:32:54.473 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:32:54.474 INFO:teuthology.packaging:ref: None 2023-11-15T06:32:54.474 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:32:54.474 INFO:teuthology.packaging:branch: None 2023-11-15T06:32:54.474 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:32:54.858 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:32:54.858 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:32:55.132 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T06:32:55.133 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T06:32:55.139 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2023-11-15T06:32:55.139 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2023-11-15T06:32:55.192 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T06:32:55.193 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:32:55.193 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T06:32:55.198 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2023-11-15T06:32:55.198 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:32:55.199 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/etc/yum.repos.d/ceph.repo 2023-11-15T06:32:55.234 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T06:32:55.235 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:32:55.236 INFO:teuthology.packaging:ref: None 2023-11-15T06:32:55.237 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:32:55.237 INFO:teuthology.packaging:branch: None 2023-11-15T06:32:55.238 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:32:55.238 DEBUG:teuthology.orchestra.run.smithi123:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T06:32:55.242 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2023-11-15T06:32:55.242 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:32:55.243 INFO:teuthology.packaging:ref: None 2023-11-15T06:32:55.244 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:32:55.244 INFO:teuthology.packaging:branch: None 2023-11-15T06:32:55.244 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:32:55.245 DEBUG:teuthology.orchestra.run.smithi135:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2023-11-15T06:32:55.306 DEBUG:teuthology.orchestra.run.smithi123:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T06:32:55.316 DEBUG:teuthology.orchestra.run.smithi135:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2023-11-15T06:32:55.384 DEBUG:teuthology.orchestra.run.smithi123:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T06:32:55.396 DEBUG:teuthology.orchestra.run.smithi135:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2023-11-15T06:32:55.450 INFO:teuthology.orchestra.run.smithi123.stdout:check_obsoletes = 1 2023-11-15T06:32:55.451 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum clean all 2023-11-15T06:32:55.464 INFO:teuthology.orchestra.run.smithi135.stdout:check_obsoletes = 1 2023-11-15T06:32:55.466 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum clean all 2023-11-15T06:32:55.721 INFO:teuthology.orchestra.run.smithi123.stdout:76 files removed 2023-11-15T06:32:55.742 INFO:teuthology.orchestra.run.smithi135.stdout:76 files removed 2023-11-15T06:32:55.748 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-radosgw 2023-11-15T06:32:55.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-radosgw 2023-11-15T06:32:56.230 INFO:teuthology.orchestra.run.smithi123.stdout:ceph packages for x86_64 962 kB/s | 74 kB 00:00 2023-11-15T06:32:56.259 INFO:teuthology.orchestra.run.smithi135.stdout:ceph packages for x86_64 919 kB/s | 74 kB 00:00 2023-11-15T06:32:56.339 INFO:teuthology.orchestra.run.smithi123.stdout:ceph noarch packages 211 kB/s | 15 kB 00:00 2023-11-15T06:32:56.376 INFO:teuthology.orchestra.run.smithi135.stdout:ceph noarch packages 191 kB/s | 15 kB 00:00 2023-11-15T06:32:56.440 INFO:teuthology.orchestra.run.smithi123.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-11-15T06:32:56.477 INFO:teuthology.orchestra.run.smithi135.stdout:ceph source packages 24 kB/s | 1.7 kB 00:00 2023-11-15T06:32:57.698 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - AppStream 28 MB/s | 34 MB 00:01 2023-11-15T06:32:59.402 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - AppStream 12 MB/s | 34 MB 00:02 2023-11-15T06:33:06.191 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - BaseOS 23 MB/s | 53 MB 00:02 2023-11-15T06:33:07.879 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - BaseOS 25 MB/s | 53 MB 00:02 2023-11-15T06:33:14.149 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras 69 kB/s | 18 kB 00:00 2023-11-15T06:33:14.549 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2023-11-15T06:33:15.542 INFO:teuthology.orchestra.run.smithi123.stdout:CentOS Stream 8 - PowerTools 7.9 MB/s | 6.6 MB 00:00 2023-11-15T06:33:15.845 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras 123 kB/s | 18 kB 00:00 2023-11-15T06:33:16.271 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - Extras common packages 28 kB/s | 6.9 kB 00:00 2023-11-15T06:33:16.994 INFO:teuthology.orchestra.run.smithi123.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-15T06:33:18.111 INFO:teuthology.orchestra.run.smithi123.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2023-11-15T06:33:18.747 INFO:teuthology.orchestra.run.smithi135.stdout:CentOS Stream 8 - PowerTools 2.8 MB/s | 6.6 MB 00:02 2023-11-15T06:33:20.198 INFO:teuthology.orchestra.run.smithi135.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2023-11-15T06:33:21.719 INFO:teuthology.orchestra.run.smithi135.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2023-11-15T06:33:22.062 INFO:teuthology.orchestra.run.smithi123.stdout:lab-extras 448 kB/s | 24 kB 00:00 2023-11-15T06:33:25.625 INFO:teuthology.orchestra.run.smithi135.stdout:lab-extras 416 kB/s | 24 kB 00:00 2023-11-15T06:33:26.815 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:33:26.817 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:33:26.818 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-15T06:33:26.818 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:33:26.818 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:33:26.818 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T06:33:26.818 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrading: 2023-11-15T06:33:26.819 INFO:teuthology.orchestra.run.smithi123.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:33:26.819 INFO:teuthology.orchestra.run.smithi123.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T06:33:26.819 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:33:26.819 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T06:33:26.819 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T06:33:26.820 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T06:33:26.820 INFO:teuthology.orchestra.run.smithi123.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T06:33:26.820 INFO:teuthology.orchestra.run.smithi123.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T06:33:26.820 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T06:33:26.820 INFO:teuthology.orchestra.run.smithi123.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T06:33:26.821 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T06:33:26.821 INFO:teuthology.orchestra.run.smithi123.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T06:33:26.821 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T06:33:26.821 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T06:33:26.821 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T06:33:26.822 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T06:33:26.822 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T06:33:26.822 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T06:33:26.822 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T06:33:26.822 INFO:teuthology.orchestra.run.smithi123.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T06:33:26.823 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T06:33:26.823 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T06:33:26.823 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T06:33:26.823 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T06:33:26.824 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T06:33:26.824 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T06:33:26.824 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T06:33:26.824 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:33:26.824 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:33:26.825 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:33:26.825 INFO:teuthology.orchestra.run.smithi123.stdout:Install 25 Packages 2023-11-15T06:33:26.825 INFO:teuthology.orchestra.run.smithi123.stdout:Upgrade 2 Packages 2023-11-15T06:33:26.825 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:33:26.826 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 59 M 2023-11-15T06:33:26.826 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:33:27.554 INFO:teuthology.orchestra.run.smithi123.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 45 MB/s | 6.0 MB 00:00 2023-11-15T06:33:27.579 INFO:teuthology.orchestra.run.smithi123.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 996 kB/s | 25 kB 00:00 2023-11-15T06:33:27.613 INFO:teuthology.orchestra.run.smithi123.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2023-11-15T06:33:27.647 INFO:teuthology.orchestra.run.smithi123.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2023-11-15T06:33:27.747 INFO:teuthology.orchestra.run.smithi123.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 41 MB/s | 4.1 MB 00:00 2023-11-15T06:33:27.772 INFO:teuthology.orchestra.run.smithi123.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.8 MB/s | 44 kB 00:00 2023-11-15T06:33:27.798 INFO:teuthology.orchestra.run.smithi123.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T06:33:27.823 INFO:teuthology.orchestra.run.smithi123.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T06:33:27.857 INFO:teuthology.orchestra.run.smithi123.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T06:33:27.891 INFO:teuthology.orchestra.run.smithi123.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 11 MB/s | 368 kB 00:00 2023-11-15T06:33:27.915 INFO:teuthology.orchestra.run.smithi123.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2023-11-15T06:33:28.066 INFO:teuthology.orchestra.run.smithi123.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2023-11-15T06:33:28.325 INFO:teuthology.orchestra.run.smithi123.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 27 MB/s | 24 MB 00:00 2023-11-15T06:33:28.350 INFO:teuthology.orchestra.run.smithi123.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 335 kB/s | 145 kB 00:00 2023-11-15T06:33:28.408 INFO:teuthology.orchestra.run.smithi123.stdout:(15/27): librdkafka-0.11.4-3.el8.x86_64.rpm 1.0 MB/s | 354 kB 00:00 2023-11-15T06:33:28.434 INFO:teuthology.orchestra.run.smithi123.stdout:(16/27): python3-prettytable-0.7.2-14.el8.noarc 524 kB/s | 44 kB 00:00 2023-11-15T06:33:28.534 INFO:teuthology.orchestra.run.smithi123.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.2 MB/s | 259 kB 00:00 2023-11-15T06:33:28.785 INFO:teuthology.orchestra.run.smithi123.stdout:(18/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 134 kB/s | 47 kB 00:00 2023-11-15T06:33:28.810 INFO:teuthology.orchestra.run.smithi123.stdout:(19/27): daxctl-libs-71.1-7.el8.x86_64.rpm 105 kB/s | 42 kB 00:00 2023-11-15T06:33:28.860 INFO:teuthology.orchestra.run.smithi123.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 246 kB/s | 80 kB 00:00 2023-11-15T06:33:28.985 INFO:teuthology.orchestra.run.smithi123.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 581 kB/s | 116 kB 00:00 2023-11-15T06:33:29.035 INFO:teuthology.orchestra.run.smithi123.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 1.3 MB/s | 306 kB 00:00 2023-11-15T06:33:29.061 INFO:teuthology.orchestra.run.smithi123.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 902 kB/s | 181 kB 00:00 2023-11-15T06:33:29.086 INFO:teuthology.orchestra.run.smithi123.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 586 kB/s | 59 kB 00:00 2023-11-15T06:33:29.111 INFO:teuthology.orchestra.run.smithi123.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 999 kB/s | 75 kB 00:00 2023-11-15T06:33:29.203 INFO:teuthology.orchestra.run.smithi123.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-11-15T06:33:29.262 INFO:teuthology.orchestra.run.smithi123.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 23 MB/s | 4.0 MB 00:00 2023-11-15T06:33:29.264 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:33:29.264 INFO:teuthology.orchestra.run.smithi123.stdout:Total 24 MB/s | 59 MB 00:02 2023-11-15T06:33:29.328 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:33:29.376 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:33:29.377 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:33:29.724 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:33:29.727 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:33:30.149 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:33:30.152 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:33:30.152 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-11-15T06:33:30.152 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:33:30.153 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:33:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2023-11-15T06:33:30.153 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrading: 2023-11-15T06:33:30.153 INFO:teuthology.orchestra.run.smithi135.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:33:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2023-11-15T06:33:30.154 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:33:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2023-11-15T06:33:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2023-11-15T06:33:30.154 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2023-11-15T06:33:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2023-11-15T06:33:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2023-11-15T06:33:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2023-11-15T06:33:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2023-11-15T06:33:30.155 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2023-11-15T06:33:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2023-11-15T06:33:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2023-11-15T06:33:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq x86_64 0.9.0-4.el8 baseos 47 k 2023-11-15T06:33:30.156 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2023-11-15T06:33:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka x86_64 0.11.4-3.el8 appstream 354 k 2023-11-15T06:33:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2023-11-15T06:33:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2023-11-15T06:33:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2023-11-15T06:33:30.157 INFO:teuthology.orchestra.run.smithi135.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2023-11-15T06:33:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2023-11-15T06:33:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2023-11-15T06:33:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2023-11-15T06:33:30.158 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2023-11-15T06:33:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2023-11-15T06:33:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2023-11-15T06:33:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2023-11-15T06:33:30.159 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:33:30.159 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:33:30.160 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:33:30.160 INFO:teuthology.orchestra.run.smithi135.stdout:Install 25 Packages 2023-11-15T06:33:30.160 INFO:teuthology.orchestra.run.smithi135.stdout:Upgrade 2 Packages 2023-11-15T06:33:30.160 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:33:30.161 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 59 M 2023-11-15T06:33:30.162 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:33:30.933 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:33:31.035 INFO:teuthology.orchestra.run.smithi135.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 37 MB/s | 6.0 MB 00:00 2023-11-15T06:33:31.060 INFO:teuthology.orchestra.run.smithi135.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 997 kB/s | 25 kB 00:00 2023-11-15T06:33:31.102 INFO:teuthology.orchestra.run.smithi135.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 19 MB/s | 801 kB 00:00 2023-11-15T06:33:31.144 INFO:teuthology.orchestra.run.smithi135.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 11 MB/s | 486 kB 00:00 2023-11-15T06:33:31.145 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T06:33:31.253 INFO:teuthology.orchestra.run.smithi135.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 38 MB/s | 4.1 MB 00:00 2023-11-15T06:33:31.278 INFO:teuthology.orchestra.run.smithi135.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 1.7 MB/s | 44 kB 00:00 2023-11-15T06:33:31.303 INFO:teuthology.orchestra.run.smithi135.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2023-11-15T06:33:31.329 INFO:teuthology.orchestra.run.smithi135.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2023-11-15T06:33:31.362 INFO:teuthology.orchestra.run.smithi135.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2023-11-15T06:33:31.374 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T06:33:31.426 INFO:teuthology.orchestra.run.smithi135.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 5.6 MB/s | 368 kB 00:00 2023-11-15T06:33:31.454 INFO:teuthology.orchestra.run.smithi135.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 114 kB 00:00 2023-11-15T06:33:31.533 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:33:31.580 INFO:teuthology.orchestra.run.smithi135.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 18 MB/s | 13 MB 00:00 2023-11-15T06:33:31.880 INFO:teuthology.orchestra.run.smithi135.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:01 2023-11-15T06:33:31.906 INFO:teuthology.orchestra.run.smithi135.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 322 kB/s | 145 kB 00:00 2023-11-15T06:33:31.929 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:33:31.956 INFO:teuthology.orchestra.run.smithi135.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 877 kB/s | 44 kB 00:00 2023-11-15T06:33:31.981 INFO:teuthology.orchestra.run.smithi135.stdout:(16/27): librdkafka-0.11.4-3.el8.x86_64.rpm 882 kB/s | 354 kB 00:00 2023-11-15T06:33:32.082 INFO:teuthology.orchestra.run.smithi135.stdout:(17/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 1.3 MB/s | 259 kB 00:00 2023-11-15T06:33:32.141 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:33:32.220 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:33:32.332 INFO:teuthology.orchestra.run.smithi135.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 112 kB/s | 42 kB 00:00 2023-11-15T06:33:32.357 INFO:teuthology.orchestra.run.smithi135.stdout:(19/27): librabbitmq-0.9.0-4.el8.x86_64.rpm 125 kB/s | 47 kB 00:00 2023-11-15T06:33:32.407 INFO:teuthology.orchestra.run.smithi135.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 246 kB/s | 80 kB 00:00 2023-11-15T06:33:32.457 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:33:32.608 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:33:32.632 INFO:teuthology.orchestra.run.smithi135.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 423 kB/s | 116 kB 00:00 2023-11-15T06:33:32.666 INFO:teuthology.orchestra.run.smithi135.stdout:(22/27): gperftools-libs-2.7-9.el8.x86_64.rpm 991 kB/s | 306 kB 00:00 2023-11-15T06:33:32.673 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:33:32.692 INFO:teuthology.orchestra.run.smithi135.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 637 kB/s | 181 kB 00:00 2023-11-15T06:33:32.709 INFO:teuthology.orchestra.run.smithi135.stdout:(24/27): liboath-2.6.2-3.el8.x86_64.rpm 774 kB/s | 59 kB 00:00 2023-11-15T06:33:32.726 INFO:teuthology.orchestra.run.smithi135.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 1.2 MB/s | 75 kB 00:00 2023-11-15T06:33:32.810 INFO:teuthology.orchestra.run.smithi135.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 32 MB/s | 3.7 MB 00:00 2023-11-15T06:33:32.889 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T06:33:32.906 INFO:teuthology.orchestra.run.smithi135.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 21 MB/s | 4.0 MB 00:00 2023-11-15T06:33:32.908 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:33:32.909 INFO:teuthology.orchestra.run.smithi135.stdout:Total 21 MB/s | 59 MB 00:02 2023-11-15T06:33:32.975 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:33:33.024 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:33:33.024 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:33:33.066 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:33:33.377 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:33:33.381 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:33:33.471 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:33:33.607 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:33:33.681 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:33:33.879 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T06:33:34.041 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:33:34.103 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:33:34.266 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:33:34.432 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:33:34.492 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:33:34.599 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:33:34.655 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:33:34.718 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2023-11-15T06:33:34.815 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:33:34.955 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2023-11-15T06:33:35.152 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:33:35.229 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:33:35.540 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2023-11-15T06:33:35.574 INFO:teuthology.orchestra.run.smithi123.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:33:35.666 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:33:35.777 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:33:35.862 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T06:33:35.877 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:33:36.076 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T06:33:36.119 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:33:36.227 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:33:36.277 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:33:36.290 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:33:36.339 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:33:36.483 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T06:33:36.558 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librabbitmq-0.9.0-4.el8.x86_64 7/29 2023-11-15T06:33:36.674 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T06:33:36.841 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T06:33:36.960 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:33:37.074 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T06:33:37.386 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librdkafka-0.11.4-3.el8.x86_64 8/29 2023-11-15T06:33:37.576 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:33:37.655 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:33:37.823 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2023-11-15T06:33:38.043 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:33:38.114 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2023-11-15T06:33:38.290 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:33:38.477 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:33:38.684 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:33:38.756 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2023-11-15T06:33:38.901 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:33:39.333 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2023-11-15T06:33:39.559 INFO:teuthology.orchestra.run.smithi135.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:33:39.656 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:33:39.660 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2023-11-15T06:33:39.855 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2023-11-15T06:33:39.873 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:33:40.086 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2023-11-15T06:33:40.284 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:33:40.362 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2023-11-15T06:33:40.560 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2023-11-15T06:33:40.567 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:33:40.734 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2023-11-15T06:33:40.846 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:33:40.934 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:33:40.949 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2023-11-15T06:33:41.056 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:33:41.184 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2023-11-15T06:33:43.733 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:33:43.934 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:33:44.628 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2023-11-15T06:33:44.924 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:33:45.024 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2023-11-15T06:33:45.242 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:33:59.241 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:33:59.242 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /sys 2023-11-15T06:33:59.242 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /proc 2023-11-15T06:33:59.242 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-11-15T06:33:59.242 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /run 2023-11-15T06:33:59.243 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /mnt 2023-11-15T06:33:59.243 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /var/tmp 2023-11-15T06:33:59.243 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /home 2023-11-15T06:33:59.243 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /tmp 2023-11-15T06:33:59.243 INFO:teuthology.orchestra.run.smithi123.stdout:skipping the directory /dev 2023-11-15T06:33:59.244 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:00.576 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:34:00.644 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:34:00.644 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:34:00.880 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:34:00.881 INFO:teuthology.orchestra.run.smithi123.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:34:01.561 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:34:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T06:34:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T06:34:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T06:34:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:34:01.562 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:34:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:34:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T06:34:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T06:34:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:34:01.563 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T06:34:01.564 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T06:34:01.564 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:34:01.564 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:34:01.564 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T06:34:01.564 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T06:34:01.565 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T06:34:01.565 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T06:34:01.565 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T06:34:01.565 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T06:34:01.565 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T06:34:01.566 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T06:34:01.566 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T06:34:01.566 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T06:34:01.566 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T06:34:01.566 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T06:34:01.567 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:34:01.567 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T06:34:01.567 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T06:34:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:34:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:02.664 INFO:teuthology.orchestra.run.smithi123.stdout:Upgraded: 2023-11-15T06:34:02.664 INFO:teuthology.orchestra.run.smithi123.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.665 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:34:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.665 INFO:teuthology.orchestra.run.smithi123.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T06:34:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T06:34:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T06:34:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T06:34:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.666 INFO:teuthology.orchestra.run.smithi123.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T06:34:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T06:34:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T06:34:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T06:34:02.667 INFO:teuthology.orchestra.run.smithi123.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T06:34:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T06:34:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T06:34:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.668 INFO:teuthology.orchestra.run.smithi123.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T06:34:02.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.669 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:02.669 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:02.670 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:02.863 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-test 2023-11-15T06:34:03.716 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:41 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:03.821 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:34:03.822 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /sys 2023-11-15T06:34:03.822 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /proc 2023-11-15T06:34:03.822 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /dev 2023-11-15T06:34:03.822 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /run 2023-11-15T06:34:03.822 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /mnt 2023-11-15T06:34:03.823 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /var/tmp 2023-11-15T06:34:03.823 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /home 2023-11-15T06:34:03.823 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /tmp 2023-11-15T06:34:03.824 INFO:teuthology.orchestra.run.smithi135.stdout:skipping the directory /dev 2023-11-15T06:34:03.824 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:05.145 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:34:05.223 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2023-11-15T06:34:05.223 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:34:05.364 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:05.365 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:05.365 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:34:05.365 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:05.366 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:34:05.366 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T06:34:05.366 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:34:05.366 INFO:teuthology.orchestra.run.smithi123.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T06:34:05.366 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T06:34:05.367 INFO:teuthology.orchestra.run.smithi123.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T06:34:05.367 INFO:teuthology.orchestra.run.smithi123.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T06:34:05.367 INFO:teuthology.orchestra.run.smithi123.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T06:34:05.367 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:05.367 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:34:05.368 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:05.368 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2023-11-15T06:34:05.368 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:05.368 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 53 M 2023-11-15T06:34:05.368 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 229 M 2023-11-15T06:34:05.369 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:34:05.441 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2023-11-15T06:34:05.442 INFO:teuthology.orchestra.run.smithi135.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:34:05.476 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 169 kB 00:00 2023-11-15T06:34:06.151 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:34:06.151 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2023-11-15T06:34:06.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2023-11-15T06:34:06.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2023-11-15T06:34:06.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2023-11-15T06:34:06.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2023-11-15T06:34:06.152 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2023-11-15T06:34:06.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2023-11-15T06:34:06.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2023-11-15T06:34:06.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2023-11-15T06:34:06.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2023-11-15T06:34:06.153 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2023-11-15T06:34:06.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2023-11-15T06:34:06.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2023-11-15T06:34:06.154 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2023-11-15T06:34:06.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librdkafka-0.11.4-3.el8.x86_64 15/29 2023-11-15T06:34:06.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2023-11-15T06:34:06.155 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2023-11-15T06:34:06.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2023-11-15T06:34:06.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librabbitmq-0.9.0-4.el8.x86_64 19/29 2023-11-15T06:34:06.156 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2023-11-15T06:34:06.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2023-11-15T06:34:06.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2023-11-15T06:34:06.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2023-11-15T06:34:06.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2023-11-15T06:34:06.157 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2023-11-15T06:34:06.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2023-11-15T06:34:06.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2023-11-15T06:34:06.158 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2023-11-15T06:34:06.162 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 70 MB/s | 52 MB 00:00 2023-11-15T06:34:06.176 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 264 kB/s | 203 kB 00:00 2023-11-15T06:34:06.202 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): oniguruma-6.8.2-2.el8.x86_64.rpm 258 kB/s | 187 kB 00:00 2023-11-15T06:34:06.294 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 606 kB/s | 71 kB 00:00 2023-11-15T06:34:06.502 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 922 kB/s | 323 kB 00:00 2023-11-15T06:34:06.503 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:06.503 INFO:teuthology.orchestra.run.smithi123.stdout:Total 46 MB/s | 53 MB 00:01 2023-11-15T06:34:06.546 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:34:06.569 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:34:06.569 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:34:06.872 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:34:06.875 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:34:07.363 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2023-11-15T06:34:07.364 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:07.364 INFO:teuthology.orchestra.run.smithi135.stdout:Upgraded: 2023-11-15T06:34:07.364 INFO:teuthology.orchestra.run.smithi135.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.364 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:34:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.365 INFO:teuthology.orchestra.run.smithi135.stdout: daxctl-libs-71.1-7.el8.x86_64 2023-11-15T06:34:07.366 INFO:teuthology.orchestra.run.smithi135.stdout: fmt-6.2.1-1.el8.x86_64 2023-11-15T06:34:07.366 INFO:teuthology.orchestra.run.smithi135.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2023-11-15T06:34:07.366 INFO:teuthology.orchestra.run.smithi135.stdout: leveldb-1.22-1.el8.x86_64 2023-11-15T06:34:07.366 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.366 INFO:teuthology.orchestra.run.smithi135.stdout: liboath-2.6.2-3.el8.x86_64 2023-11-15T06:34:07.367 INFO:teuthology.orchestra.run.smithi135.stdout: libpmemobj-1.6.1-1.el8.x86_64 2023-11-15T06:34:07.367 INFO:teuthology.orchestra.run.smithi135.stdout: librabbitmq-0.9.0-4.el8.x86_64 2023-11-15T06:34:07.367 INFO:teuthology.orchestra.run.smithi135.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.367 INFO:teuthology.orchestra.run.smithi135.stdout: librdkafka-0.11.4-3.el8.x86_64 2023-11-15T06:34:07.367 INFO:teuthology.orchestra.run.smithi135.stdout: librgw2-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.368 INFO:teuthology.orchestra.run.smithi135.stdout: libunwind-1.3.1-3.el8.x86_64 2023-11-15T06:34:07.368 INFO:teuthology.orchestra.run.smithi135.stdout: lttng-ust-2.8.1-11.el8.x86_64 2023-11-15T06:34:07.368 INFO:teuthology.orchestra.run.smithi135.stdout: ndctl-libs-71.1-7.el8.x86_64 2023-11-15T06:34:07.368 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.368 INFO:teuthology.orchestra.run.smithi135.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.369 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.369 INFO:teuthology.orchestra.run.smithi135.stdout: python3-prettytable-0.7.2-14.el8.noarch 2023-11-15T06:34:07.369 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.369 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.369 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:07.370 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:07.370 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:34:07.383 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:34:07.570 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-test 2023-11-15T06:34:07.613 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T06:34:07.804 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T06:34:07.977 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:34:08.056 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:34:08.242 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T06:34:08.417 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:43 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:34:08.427 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:34:10.068 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:34:10.069 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:10.069 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:34:10.069 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:10.069 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:34:10.070 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2023-11-15T06:34:10.070 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:34:10.070 INFO:teuthology.orchestra.run.smithi135.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2023-11-15T06:34:10.070 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2023-11-15T06:34:10.070 INFO:teuthology.orchestra.run.smithi135.stdout: oniguruma x86_64 6.8.2-2.el8 appstream 187 k 2023-11-15T06:34:10.071 INFO:teuthology.orchestra.run.smithi135.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2023-11-15T06:34:10.071 INFO:teuthology.orchestra.run.smithi135.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2023-11-15T06:34:10.071 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:10.071 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:34:10.071 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:10.071 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2023-11-15T06:34:10.072 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:10.072 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 53 M 2023-11-15T06:34:10.073 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 229 M 2023-11-15T06:34:10.073 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:34:11.332 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 169 kB 00:00 2023-11-15T06:34:12.049 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 67 MB/s | 52 MB 00:00 2023-11-15T06:34:12.074 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): oniguruma-6.8.2-2.el8.x86_64.rpm 253 kB/s | 187 kB 00:00 2023-11-15T06:34:12.100 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): jq-1.6-8.el8.x86_64.rpm 246 kB/s | 203 kB 00:00 2023-11-15T06:34:12.191 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 604 kB/s | 71 kB 00:00 2023-11-15T06:34:12.349 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 1.1 MB/s | 323 kB 00:00 2023-11-15T06:34:12.351 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:12.351 INFO:teuthology.orchestra.run.smithi135.stdout:Total 23 MB/s | 53 MB 00:02 2023-11-15T06:34:12.394 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:34:12.414 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:34:12.415 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:34:12.724 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:34:12.727 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:34:13.227 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:34:13.290 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:34:13.449 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2023-11-15T06:34:13.517 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:34:13.648 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2023-11-15T06:34:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:34:13.814 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T06:34:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T06:34:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T06:34:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T06:34:13.815 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T06:34:13.821 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:34:13.887 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: oniguruma-6.8.2-2.el8.x86_64 3/6 2023-11-15T06:34:14.070 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2023-11-15T06:34:14.271 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:34:14.505 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T06:34:14.505 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:14.506 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:34:14.506 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T06:34:14.506 INFO:teuthology.orchestra.run.smithi123.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T06:34:14.506 INFO:teuthology.orchestra.run.smithi123.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T06:34:14.507 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:14.507 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:14.737 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph 2023-11-15T06:34:15.595 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:00:53 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:17.266 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:17.270 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:17.270 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-15T06:34:17.270 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:17.270 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:34:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T06:34:17.271 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:34:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T06:34:17.271 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T06:34:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T06:34:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T06:34:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T06:34:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T06:34:17.272 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T06:34:17.273 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T06:34:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T06:34:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T06:34:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T06:34:17.274 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T06:34:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T06:34:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T06:34:17.275 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T06:34:17.276 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T06:34:17.277 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T06:34:17.278 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:34:17.279 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:17.280 INFO:teuthology.orchestra.run.smithi123.stdout:Install 32 Packages 2023-11-15T06:34:17.280 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:17.281 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 31 M 2023-11-15T06:34:17.281 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 111 M 2023-11-15T06:34:17.281 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:34:18.195 INFO:teuthology.orchestra.run.smithi123.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 121 kB/s | 6.3 kB 00:00 2023-11-15T06:34:18.254 INFO:teuthology.orchestra.run.smithi123.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2023-11-15T06:34:18.312 INFO:teuthology.orchestra.run.smithi123.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2023-11-15T06:34:18.372 INFO:teuthology.orchestra.run.smithi123.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 3.6 MB/s | 220 kB 00:00 2023-11-15T06:34:18.446 INFO:teuthology.orchestra.run.smithi123.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2023-11-15T06:34:18.671 INFO:teuthology.orchestra.run.smithi123.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2023-11-15T06:34:18.780 INFO:teuthology.orchestra.run.smithi123.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 385 kB/s | 157 kB 00:00 2023-11-15T06:34:18.805 INFO:teuthology.orchestra.run.smithi123.stdout:(8/32): python3-markupsafe-0.23-19.el8.x86_64.r 108 kB/s | 39 kB 00:00 2023-11-15T06:34:18.855 INFO:teuthology.orchestra.run.smithi123.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 760 kB/s | 57 kB 00:00 2023-11-15T06:34:18.988 INFO:teuthology.orchestra.run.smithi123.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 326 kB/s | 103 kB 00:00 2023-11-15T06:34:19.133 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2023-11-15T06:34:19.256 INFO:teuthology.orchestra.run.smithi123.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.0 MB/s | 457 kB 00:00 2023-11-15T06:34:19.393 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:34:19.523 INFO:teuthology.orchestra.run.smithi123.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 373 kB/s | 249 kB 00:00 2023-11-15T06:34:19.679 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2023-11-15T06:34:19.679 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2023-11-15T06:34:19.679 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2023-11-15T06:34:19.679 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2023-11-15T06:34:19.680 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : oniguruma-6.8.2-2.el8.x86_64 4/6 2023-11-15T06:34:19.680 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2023-11-15T06:34:19.756 INFO:teuthology.orchestra.run.smithi123.stdout:(13/32): python3-cryptography-3.2.1-6.el8.x86_6 836 kB/s | 641 kB 00:00 2023-11-15T06:34:19.790 INFO:teuthology.orchestra.run.smithi123.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 204 kB/s | 109 kB 00:00 2023-11-15T06:34:20.032 INFO:teuthology.orchestra.run.smithi123.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 86 kB/s | 44 kB 00:00 2023-11-15T06:34:20.392 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2023-11-15T06:34:20.393 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:20.393 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:34:20.393 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2023-11-15T06:34:20.393 INFO:teuthology.orchestra.run.smithi135.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-2.el8.x86_64 2023-11-15T06:34:20.393 INFO:teuthology.orchestra.run.smithi135.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2023-11-15T06:34:20.394 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:20.394 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:34:20.432 INFO:teuthology.orchestra.run.smithi123.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 274 kB/s | 185 kB 00:00 2023-11-15T06:34:20.458 INFO:teuthology.orchestra.run.smithi123.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 259 kB/s | 173 kB 00:00 2023-11-15T06:34:20.491 INFO:teuthology.orchestra.run.smithi123.stdout:(18/32): python3-cherrypy-18.4.0-1.el8.noarch.r 838 kB/s | 384 kB 00:00 2023-11-15T06:34:20.558 INFO:teuthology.orchestra.run.smithi123.stdout:(19/32): python3-jaraco-6.2-6.el8.noarch.rpm 88 kB/s | 11 kB 00:00 2023-11-15T06:34:20.591 INFO:teuthology.orchestra.run.smithi123.stdout:(20/32): python3-jaraco-functools-2.0-4.el8.noa 131 kB/s | 18 kB 00:00 2023-11-15T06:34:20.616 INFO:teuthology.orchestra.run.smithi123.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 388 kB/s | 49 kB 00:00 2023-11-15T06:34:20.635 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph 2023-11-15T06:34:20.700 INFO:teuthology.orchestra.run.smithi123.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 413 kB/s | 59 kB 00:00 2023-11-15T06:34:20.734 INFO:teuthology.orchestra.run.smithi123.stdout:(23/32): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2023-11-15T06:34:20.834 INFO:teuthology.orchestra.run.smithi123.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 140 kB/s | 19 kB 00:00 2023-11-15T06:34:20.859 INFO:teuthology.orchestra.run.smithi123.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2023-11-15T06:34:20.893 INFO:teuthology.orchestra.run.smithi123.stdout:(26/32): python3-pecan-1.3.2-9.el8.noarch.rpm 938 kB/s | 283 kB 00:00 2023-11-15T06:34:20.968 INFO:teuthology.orchestra.run.smithi123.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 218 kB/s | 29 kB 00:00 2023-11-15T06:34:20.994 INFO:teuthology.orchestra.run.smithi123.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 205 kB/s | 27 kB 00:00 2023-11-15T06:34:21.061 INFO:teuthology.orchestra.run.smithi123.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.5 MB/s | 255 kB 00:00 2023-11-15T06:34:21.128 INFO:teuthology.orchestra.run.smithi123.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 639 kB/s | 85 kB 00:00 2023-11-15T06:34:21.186 INFO:teuthology.orchestra.run.smithi123.stdout:(31/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.1 MB/s | 251 kB 00:00 2023-11-15T06:34:21.211 INFO:teuthology.orchestra.run.smithi123.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 153 kB/s | 23 kB 00:00 2023-11-15T06:34:21.215 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:21.216 INFO:teuthology.orchestra.run.smithi123.stdout:Total 7.9 MB/s | 31 MB 00:03 2023-11-15T06:34:21.297 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:34:21.336 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:34:21.337 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:34:21.495 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:00:56 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:34:21.641 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:34:21.644 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:34:22.735 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:34:22.921 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T06:34:23.132 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T06:34:23.153 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:34:23.156 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:23.157 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-11-15T06:34:23.157 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:23.157 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:34:23.157 INFO:teuthology.orchestra.run.smithi135.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2023-11-15T06:34:23.157 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:34:23.158 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2023-11-15T06:34:23.158 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2023-11-15T06:34:23.158 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2023-11-15T06:34:23.158 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2023-11-15T06:34:23.158 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2023-11-15T06:34:23.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2023-11-15T06:34:23.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2023-11-15T06:34:23.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2023-11-15T06:34:23.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2023-11-15T06:34:23.159 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2023-11-15T06:34:23.160 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography x86_64 3.2.1-6.el8 baseos 641 k 2023-11-15T06:34:23.160 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2023-11-15T06:34:23.160 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2023-11-15T06:34:23.160 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2023-11-15T06:34:23.160 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2023-11-15T06:34:23.161 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2023-11-15T06:34:23.161 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2023-11-15T06:34:23.161 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2023-11-15T06:34:23.161 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2023-11-15T06:34:23.161 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2023-11-15T06:34:23.162 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2023-11-15T06:34:23.162 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2023-11-15T06:34:23.162 INFO:teuthology.orchestra.run.smithi135.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2023-11-15T06:34:23.162 INFO:teuthology.orchestra.run.smithi135.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2023-11-15T06:34:23.162 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2023-11-15T06:34:23.163 INFO:teuthology.orchestra.run.smithi135.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2023-11-15T06:34:23.163 INFO:teuthology.orchestra.run.smithi135.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2023-11-15T06:34:23.163 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2023-11-15T06:34:23.163 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2023-11-15T06:34:23.163 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2023-11-15T06:34:23.164 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2023-11-15T06:34:23.164 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:23.164 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:34:23.164 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:23.164 INFO:teuthology.orchestra.run.smithi135.stdout:Install 32 Packages 2023-11-15T06:34:23.165 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:23.165 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 31 M 2023-11-15T06:34:23.166 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 111 M 2023-11-15T06:34:23.166 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:34:23.374 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T06:34:23.502 INFO:teuthology.orchestra.run.smithi135.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 115 kB/s | 6.3 kB 00:00 2023-11-15T06:34:23.552 INFO:teuthology.orchestra.run.smithi135.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 15 MB/s | 1.6 MB 00:00 2023-11-15T06:34:23.555 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T06:34:23.603 INFO:teuthology.orchestra.run.smithi135.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 15 MB/s | 2.4 MB 00:00 2023-11-15T06:34:23.636 INFO:teuthology.orchestra.run.smithi135.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 6.5 MB/s | 220 kB 00:00 2023-11-15T06:34:23.743 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T06:34:23.754 INFO:teuthology.orchestra.run.smithi135.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 20 MB/s | 5.0 MB 00:00 2023-11-15T06:34:23.917 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T06:34:24.021 INFO:teuthology.orchestra.run.smithi135.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 39 MB/s | 18 MB 00:00 2023-11-15T06:34:24.038 INFO:teuthology.orchestra.run.smithi135.stdout:(7/32): python3-markupsafe-0.23-19.el8.x86_64.r 137 kB/s | 39 kB 00:00 2023-11-15T06:34:24.063 INFO:teuthology.orchestra.run.smithi135.stdout:(8/32): python3-mako-1.0.6-14.el8.noarch.rpm 368 kB/s | 157 kB 00:00 2023-11-15T06:34:24.068 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T06:34:24.161 INFO:teuthology.orchestra.run.smithi135.stdout:(9/32): python3-pytz-2017.2-11.el8.noarch.rpm 464 kB/s | 57 kB 00:00 2023-11-15T06:34:24.229 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T06:34:24.230 INFO:teuthology.orchestra.run.smithi135.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 495 kB/s | 103 kB 00:00 2023-11-15T06:34:24.322 INFO:teuthology.orchestra.run.smithi135.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.7 MB/s | 457 kB 00:00 2023-11-15T06:34:24.413 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T06:34:24.651 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T06:34:24.714 INFO:teuthology.orchestra.run.smithi135.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 450 kB/s | 249 kB 00:00 2023-11-15T06:34:24.789 INFO:teuthology.orchestra.run.smithi135.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 233 kB/s | 109 kB 00:00 2023-11-15T06:34:24.867 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T06:34:24.890 INFO:teuthology.orchestra.run.smithi135.stdout:(14/32): python3-cryptography-3.2.1-6.el8.x86_6 973 kB/s | 641 kB 00:00 2023-11-15T06:34:25.096 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T06:34:25.223 INFO:teuthology.orchestra.run.smithi135.stdout:(15/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 86 kB/s | 44 kB 00:00 2023-11-15T06:34:25.449 INFO:teuthology.orchestra.run.smithi135.stdout:(16/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 281 kB/s | 185 kB 00:00 2023-11-15T06:34:25.487 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T06:34:25.490 INFO:teuthology.orchestra.run.smithi135.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 287 kB/s | 173 kB 00:00 2023-11-15T06:34:25.583 INFO:teuthology.orchestra.run.smithi135.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 83 kB/s | 11 kB 00:00 2023-11-15T06:34:25.608 INFO:teuthology.orchestra.run.smithi135.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 150 kB/s | 18 kB 00:00 2023-11-15T06:34:25.666 INFO:teuthology.orchestra.run.smithi135.stdout:(20/32): python3-cherrypy-18.4.0-1.el8.noarch.r 868 kB/s | 384 kB 00:00 2023-11-15T06:34:25.700 INFO:teuthology.orchestra.run.smithi135.stdout:(21/32): python3-logutils-0.3.5-11.el8.noarch.r 416 kB/s | 49 kB 00:00 2023-11-15T06:34:25.725 INFO:teuthology.orchestra.run.smithi135.stdout:(22/32): python3-more-itertools-7.2.0-3.el8.noa 500 kB/s | 59 kB 00:00 2023-11-15T06:34:25.760 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T06:34:25.792 INFO:teuthology.orchestra.run.smithi135.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 2.2 MB/s | 283 kB 00:00 2023-11-15T06:34:25.818 INFO:teuthology.orchestra.run.smithi135.stdout:(24/32): python3-portend-2.6-1.el8.noarch.rpm 133 kB/s | 16 kB 00:00 2023-11-15T06:34:25.843 INFO:teuthology.orchestra.run.smithi135.stdout:(25/32): python3-simplegeneric-0.8.1-17.el8.noa 159 kB/s | 19 kB 00:00 2023-11-15T06:34:25.910 INFO:teuthology.orchestra.run.smithi135.stdout:(26/32): python3-singledispatch-3.4.0.3-18.el8. 204 kB/s | 24 kB 00:00 2023-11-15T06:34:25.915 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T06:34:25.944 INFO:teuthology.orchestra.run.smithi135.stdout:(27/32): python3-tempora-1.14.1-5.el8.noarch.rp 233 kB/s | 29 kB 00:00 2023-11-15T06:34:25.969 INFO:teuthology.orchestra.run.smithi135.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 219 kB/s | 27 kB 00:00 2023-11-15T06:34:26.036 INFO:teuthology.orchestra.run.smithi135.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 2.0 MB/s | 255 kB 00:00 2023-11-15T06:34:26.094 INFO:teuthology.orchestra.run.smithi135.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 681 kB/s | 85 kB 00:00 2023-11-15T06:34:26.161 INFO:teuthology.orchestra.run.smithi135.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 183 kB/s | 23 kB 00:00 2023-11-15T06:34:26.174 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T06:34:26.197 INFO:teuthology.orchestra.run.smithi135.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 993 kB/s | 251 kB 00:00 2023-11-15T06:34:26.201 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:26.202 INFO:teuthology.orchestra.run.smithi135.stdout:Total 10 MB/s | 31 MB 00:03 2023-11-15T06:34:26.283 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:34:26.323 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:34:26.323 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:34:26.358 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:34:26.633 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T06:34:26.646 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:34:26.649 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:34:26.846 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T06:34:27.026 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T06:34:27.223 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T06:34:27.455 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T06:34:27.716 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T06:34:27.766 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:34:27.925 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T06:34:27.974 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2023-11-15T06:34:28.185 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2023-11-15T06:34:28.237 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T06:34:28.436 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2023-11-15T06:34:28.555 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T06:34:28.642 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2023-11-15T06:34:28.823 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2023-11-15T06:34:28.925 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T06:34:28.994 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2023-11-15T06:34:29.155 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:34:29.197 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2023-11-15T06:34:29.358 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2023-11-15T06:34:29.560 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2023-11-15T06:34:29.763 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2023-11-15T06:34:30.003 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2023-11-15T06:34:30.258 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2023-11-15T06:34:30.621 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2023-11-15T06:34:30.694 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:34:30.872 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cryptography-3.2.1-6.el8.x86_64 14/32 2023-11-15T06:34:30.896 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:34:31.060 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2023-11-15T06:34:31.329 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:34:31.339 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2023-11-15T06:34:31.554 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:34:31.563 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:34:31.784 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:34:31.866 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2023-11-15T06:34:31.965 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:34:31.983 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:34:32.074 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2023-11-15T06:34:32.123 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:34:32.246 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2023-11-15T06:34:32.453 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2023-11-15T06:34:32.543 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:34:32.543 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T06:34:32.543 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T06:34:32.544 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T06:34:32.544 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T06:34:32.544 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T06:34:32.544 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T06:34:32.544 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T06:34:32.545 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T06:34:32.545 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T06:34:32.545 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T06:34:32.545 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T06:34:32.545 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T06:34:32.546 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T06:34:32.546 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T06:34:32.547 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T06:34:32.547 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T06:34:32.547 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:34:32.547 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T06:34:32.548 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T06:34:32.548 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T06:34:32.548 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T06:34:32.548 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T06:34:32.548 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T06:34:32.549 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T06:34:32.549 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T06:34:32.550 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T06:34:32.550 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T06:34:32.550 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T06:34:32.550 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T06:34:32.551 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T06:34:32.551 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T06:34:32.744 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2023-11-15T06:34:32.978 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2023-11-15T06:34:33.222 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2023-11-15T06:34:33.620 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2023-11-15T06:34:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T06:34:33.631 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:33.632 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:34:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:33.632 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T06:34:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T06:34:33.633 INFO:teuthology.orchestra.run.smithi123.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T06:34:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T06:34:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T06:34:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T06:34:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T06:34:33.634 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T06:34:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T06:34:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T06:34:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T06:34:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T06:34:33.635 INFO:teuthology.orchestra.run.smithi123.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T06:34:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T06:34:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T06:34:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T06:34:33.636 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T06:34:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T06:34:33.637 INFO:teuthology.orchestra.run.smithi123.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T06:34:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T06:34:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T06:34:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T06:34:33.638 INFO:teuthology.orchestra.run.smithi123.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T06:34:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T06:34:33.639 INFO:teuthology.orchestra.run.smithi123.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T06:34:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T06:34:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T06:34:33.640 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:33.640 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:33.868 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-base 2023-11-15T06:34:33.933 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2023-11-15T06:34:34.354 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2023-11-15T06:34:34.558 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:34:34.729 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:12 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:36.091 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2023-11-15T06:34:36.175 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:34:36.316 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:34:36.391 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:36.392 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:34:36.392 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:36.503 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install cephadm 2023-11-15T06:34:36.749 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2023-11-15T06:34:36.999 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:34:37.223 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2023-11-15T06:34:37.353 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:15 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:37.393 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:34:37.410 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2023-11-15T06:34:37.576 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:34:37.983 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2023-11-15T06:34:37.983 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2023-11-15T06:34:37.983 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2023-11-15T06:34:37.983 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2023-11-15T06:34:37.984 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2023-11-15T06:34:37.984 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2023-11-15T06:34:37.984 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2023-11-15T06:34:37.984 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2023-11-15T06:34:37.984 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2023-11-15T06:34:37.985 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2023-11-15T06:34:37.985 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2023-11-15T06:34:37.985 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2023-11-15T06:34:37.985 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2023-11-15T06:34:37.985 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cryptography-3.2.1-6.el8.x86_64 13/32 2023-11-15T06:34:37.986 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2023-11-15T06:34:37.986 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2023-11-15T06:34:37.986 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2023-11-15T06:34:37.986 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2023-11-15T06:34:37.986 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2023-11-15T06:34:37.987 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2023-11-15T06:34:37.987 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2023-11-15T06:34:37.987 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2023-11-15T06:34:37.987 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2023-11-15T06:34:37.987 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2023-11-15T06:34:37.988 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2023-11-15T06:34:37.988 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2023-11-15T06:34:37.988 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2023-11-15T06:34:37.988 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2023-11-15T06:34:37.988 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2023-11-15T06:34:37.989 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2023-11-15T06:34:37.989 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2023-11-15T06:34:37.989 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2023-11-15T06:34:39.006 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:39.007 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:39.007 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:34:39.008 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:39.008 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:34:39.008 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T06:34:39.008 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:39.009 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:34:39.009 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:39.009 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-11-15T06:34:39.009 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:39.009 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 71 k 2023-11-15T06:34:39.009 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 290 k 2023-11-15T06:34:39.010 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:34:39.057 INFO:teuthology.orchestra.run.smithi123.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.4 MB/s | 71 kB 00:00 2023-11-15T06:34:39.058 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:39.058 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.4 MB/s | 71 kB 00:00 2023-11-15T06:34:39.058 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:34:39.065 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:34:39.065 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:34:39.066 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:34:39.069 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:34:39.141 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:34:39.628 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2023-11-15T06:34:39.629 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:39.629 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:34:39.629 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:39.629 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:39.630 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:39.630 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2023-11-15T06:34:39.630 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:39.630 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:39.630 INFO:teuthology.orchestra.run.smithi135.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2023-11-15T06:34:39.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2023-11-15T06:34:39.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cffi-1.11.5-6.el8.x86_64 2023-11-15T06:34:39.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cheroot-8.5.2-1.el8.noarch 2023-11-15T06:34:39.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2023-11-15T06:34:39.631 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cryptography-3.2.1-6.el8.x86_64 2023-11-15T06:34:39.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-6.2-6.el8.noarch 2023-11-15T06:34:39.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2023-11-15T06:34:39.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-logutils-0.3.5-11.el8.noarch 2023-11-15T06:34:39.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-mako-1.0.6-14.el8.noarch 2023-11-15T06:34:39.632 INFO:teuthology.orchestra.run.smithi135.stdout: python3-markupsafe-0.23-19.el8.x86_64 2023-11-15T06:34:39.633 INFO:teuthology.orchestra.run.smithi135.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2023-11-15T06:34:39.633 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pecan-1.3.2-9.el8.noarch 2023-11-15T06:34:39.633 INFO:teuthology.orchestra.run.smithi135.stdout: python3-portend-2.6-1.el8.noarch 2023-11-15T06:34:39.633 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2023-11-15T06:34:39.633 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pycparser-2.14-14.el8.noarch 2023-11-15T06:34:39.633 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pytz-2017.2-11.el8.noarch 2023-11-15T06:34:39.634 INFO:teuthology.orchestra.run.smithi135.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2023-11-15T06:34:39.634 INFO:teuthology.orchestra.run.smithi135.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2023-11-15T06:34:39.634 INFO:teuthology.orchestra.run.smithi135.stdout: python3-tempora-1.14.1-5.el8.noarch 2023-11-15T06:34:39.634 INFO:teuthology.orchestra.run.smithi135.stdout: python3-trustme-0.6.0-4.el8.noarch 2023-11-15T06:34:39.634 INFO:teuthology.orchestra.run.smithi135.stdout: python3-waitress-1.4.3-1.el8.noarch 2023-11-15T06:34:39.635 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webob-1.8.5-1.el8.1.noarch 2023-11-15T06:34:39.635 INFO:teuthology.orchestra.run.smithi135.stdout: python3-webtest-2.0.33-1.el8.noarch 2023-11-15T06:34:39.635 INFO:teuthology.orchestra.run.smithi135.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2023-11-15T06:34:39.635 INFO:teuthology.orchestra.run.smithi135.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2023-11-15T06:34:39.635 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:39.636 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:34:39.640 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:39.975 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-base 2023-11-15T06:34:40.376 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:40.605 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:40.836 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:15 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:34:41.423 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:41.423 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:41.423 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:34:41.424 INFO:teuthology.orchestra.run.smithi123.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T06:34:41.424 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:41.424 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:41.685 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T06:34:42.285 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:34:42.501 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:34:42.502 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:34:42.503 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:34:42.545 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:20 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:42.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install cephadm 2023-11-15T06:34:43.457 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:18 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:34:44.194 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:44.196 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:44.196 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repo Size 2023-11-15T06:34:44.196 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:44.197 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:34:44.197 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T06:34:44.198 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:44.198 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:34:44.198 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:44.199 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-11-15T06:34:44.199 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:44.199 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 163 k 2023-11-15T06:34:44.200 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 587 k 2023-11-15T06:34:44.200 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:34:44.236 INFO:teuthology.orchestra.run.smithi123.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 4.0 MB/s | 163 kB 00:00 2023-11-15T06:34:44.237 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:44.237 INFO:teuthology.orchestra.run.smithi123.stdout:Total 3.8 MB/s | 163 kB 00:00 2023-11-15T06:34:44.237 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:34:44.248 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:34:44.248 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:34:44.294 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:34:44.297 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:34:44.403 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:34:44.599 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:34:45.008 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:34:45.128 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:34:45.129 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:45.129 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:34:45.129 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:45.129 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:34:45.130 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2023-11-15T06:34:45.130 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:45.130 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:34:45.130 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:45.131 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-11-15T06:34:45.131 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:45.131 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 71 k 2023-11-15T06:34:45.131 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 290 k 2023-11-15T06:34:45.131 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:34:45.206 INFO:teuthology.orchestra.run.smithi135.stdout:cephadm-16.2.4-0.el8.noarch.rpm 934 kB/s | 71 kB 00:00 2023-11-15T06:34:45.206 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:45.207 INFO:teuthology.orchestra.run.smithi135.stdout:Total 918 kB/s | 71 kB 00:00 2023-11-15T06:34:45.207 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:34:45.213 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:34:45.213 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:34:45.215 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:34:45.218 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:34:45.261 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:34:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:34:45.646 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:45.647 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:34:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:45.647 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:45.647 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:45.711 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:45.852 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr 2023-11-15T06:34:45.910 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:46.136 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:46.714 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:24 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:46.820 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2023-11-15T06:34:46.820 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:46.820 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:34:46.820 INFO:teuthology.orchestra.run.smithi135.stdout: cephadm-2:16.2.4-0.el8.noarch 2023-11-15T06:34:46.821 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:46.821 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:34:47.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-immutable-object-cache 2023-11-15T06:34:47.891 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:22 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:34:48.189 INFO:teuthology.orchestra.run.smithi123.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:34:48.401 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:48.403 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:34:48.403 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:48.524 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T06:34:49.376 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:27 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:49.558 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:34:49.559 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:49.559 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repo Size 2023-11-15T06:34:49.559 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:49.560 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:34:49.560 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2023-11-15T06:34:49.560 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:49.560 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:34:49.560 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:49.561 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-11-15T06:34:49.561 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:49.561 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 163 k 2023-11-15T06:34:49.561 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 587 k 2023-11-15T06:34:49.561 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:34:49.603 INFO:teuthology.orchestra.run.smithi135.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.7 MB/s | 163 kB 00:00 2023-11-15T06:34:49.603 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:49.603 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.6 MB/s | 163 kB 00:00 2023-11-15T06:34:49.604 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:34:49.614 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:34:49.614 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:34:49.661 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:34:49.664 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:34:49.793 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:34:49.966 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:34:50.379 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:34:51.047 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:51.048 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:51.048 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-15T06:34:51.048 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:51.048 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:34:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T06:34:51.049 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:34:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T06:34:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T06:34:51.049 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T06:34:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T06:34:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T06:34:51.050 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:51.050 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:34:51.050 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:34:51.051 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2023-11-15T06:34:51.051 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:51.051 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.0 M 2023-11-15T06:34:51.052 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 15 M 2023-11-15T06:34:51.052 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:34:51.364 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 401 kB/s | 24 kB 00:00 2023-11-15T06:34:51.389 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 105 kB/s | 8.9 kB 00:00 2023-11-15T06:34:51.448 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2023-11-15T06:34:51.465 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:34:51.465 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:51.465 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:34:51.465 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2023-11-15T06:34:51.466 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:51.466 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:34:51.506 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 287 kB/s | 34 kB 00:00 2023-11-15T06:34:51.578 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.5 MB/s | 196 kB 00:00 2023-11-15T06:34:51.715 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 121 kB/s | 43 kB 00:00 2023-11-15T06:34:51.717 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:51.718 INFO:teuthology.orchestra.run.smithi123.stdout:Total 4.4 MB/s | 3.0 MB 00:00 2023-11-15T06:34:51.728 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:34:51.738 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:34:51.738 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:34:51.920 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:34:51.923 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:34:52.197 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:34:52.417 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T06:34:52.651 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr 2023-11-15T06:34:52.654 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T06:34:52.846 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T06:34:53.000 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T06:34:53.527 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:28 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:34:54.020 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T06:34:54.283 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:34:54.553 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:34:54.553 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:34:54.553 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T06:34:54.554 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T06:34:54.554 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T06:34:54.554 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T06:34:54.975 INFO:teuthology.orchestra.run.smithi135.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:34:55.189 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:34:55.190 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:34:55.191 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:34:55.214 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T06:34:55.215 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:55.215 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:34:55.215 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T06:34:55.215 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T06:34:55.216 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T06:34:55.216 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T06:34:55.216 INFO:teuthology.orchestra.run.smithi123.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T06:34:55.216 INFO:teuthology.orchestra.run.smithi123.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T06:34:55.216 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:55.217 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:34:55.293 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-dashboard 2023-11-15T06:34:55.354 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T06:34:56.143 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:31 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:34:56.208 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:34 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:34:57.814 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:34:57.815 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:57.816 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-11-15T06:34:57.816 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:57.816 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:34:57.816 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2023-11-15T06:34:57.816 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2023-11-15T06:34:57.817 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2023-11-15T06:34:57.818 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:34:57.819 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.0 M 2023-11-15T06:34:57.819 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 15 M 2023-11-15T06:34:57.820 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:34:57.870 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:34:57.871 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================================ 2023-11-15T06:34:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-15T06:34:57.872 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================================ 2023-11-15T06:34:57.872 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:34:57.872 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T06:34:57.872 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:34:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T06:34:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T06:34:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T06:34:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T06:34:57.873 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T06:34:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T06:34:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T06:34:57.874 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:57.874 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:34:57.875 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================================ 2023-11-15T06:34:57.875 INFO:teuthology.orchestra.run.smithi123.stdout:Install 8 Packages 2023-11-15T06:34:57.875 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:34:57.876 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 37 M 2023-11-15T06:34:57.876 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 210 M 2023-11-15T06:34:57.876 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:34:58.116 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 540 kB/s | 24 kB 00:00 2023-11-15T06:34:58.142 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 128 kB/s | 8.9 kB 00:00 2023-11-15T06:34:58.217 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 18 MB/s | 2.7 MB 00:00 2023-11-15T06:34:58.254 INFO:teuthology.orchestra.run.smithi123.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 57 MB/s | 8.6 MB 00:00 2023-11-15T06:34:58.400 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 130 kB/s | 34 kB 00:00 2023-11-15T06:34:58.476 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): python3-jwt-1.6.1-2.el8.noarch.rpm 119 kB/s | 43 kB 00:00 2023-11-15T06:34:58.643 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 462 kB/s | 196 kB 00:00 2023-11-15T06:34:58.644 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:34:58.645 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.6 MB/s | 3.0 MB 00:00 2023-11-15T06:34:58.654 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:34:58.664 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:34:58.664 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:34:58.819 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:34:58.822 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:34:58.862 INFO:teuthology.orchestra.run.smithi123.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.2 MB/s | 4.7 MB 00:00 2023-11-15T06:34:58.913 INFO:teuthology.orchestra.run.smithi123.stdout:(3/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 4.4 MB/s | 225 kB 00:00 2023-11-15T06:34:58.971 INFO:teuthology.orchestra.run.smithi123.stdout:(4/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 5.2 MB/s | 3.7 MB 00:00 2023-11-15T06:34:59.093 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:34:59.130 INFO:teuthology.orchestra.run.smithi123.stdout:(5/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 4.7 MB/s | 4.9 MB 00:01 2023-11-15T06:34:59.345 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2023-11-15T06:34:59.522 INFO:teuthology.orchestra.run.smithi123.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-11-15T06:34:59.547 INFO:teuthology.orchestra.run.smithi123.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 414 kB/s | 172 kB 00:00 2023-11-15T06:34:59.598 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2023-11-15T06:34:59.799 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2023-11-15T06:35:00.003 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2023-11-15T06:35:00.122 INFO:teuthology.orchestra.run.smithi123.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 11 MB/s | 14 MB 00:01 2023-11-15T06:35:00.123 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:00.124 INFO:teuthology.orchestra.run.smithi123.stdout:Total 16 MB/s | 37 MB 00:02 2023-11-15T06:35:00.354 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:35:00.372 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:35:00.373 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:35:00.683 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:35:00.686 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:35:00.984 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2023-11-15T06:35:01.295 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:35:01.343 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:35:01.493 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:35:01.585 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:35:01.585 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:35:01.585 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2023-11-15T06:35:01.586 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2023-11-15T06:35:01.586 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2023-11-15T06:35:01.586 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2023-11-15T06:35:01.663 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:35:01.801 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:35:02.254 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2023-11-15T06:35:02.254 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:02.254 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:35:02.255 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2023-11-15T06:35:02.255 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2023-11-15T06:35:02.255 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2023-11-15T06:35:02.255 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jwt-1.6.1-2.el8.noarch 2023-11-15T06:35:02.256 INFO:teuthology.orchestra.run.smithi135.stdout: python3-repoze-lru-0.7-6.el8.noarch 2023-11-15T06:35:02.256 INFO:teuthology.orchestra.run.smithi135.stdout: python3-routes-2.4.1-12.el8.noarch 2023-11-15T06:35:02.256 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:02.256 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:02.378 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-diskprediction-local 2023-11-15T06:35:02.379 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:35:03.238 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:38 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:03.676 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:35:04.289 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:35:04.445 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:35:04.916 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:04.918 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================================ 2023-11-15T06:35:04.918 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-11-15T06:35:04.918 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================================ 2023-11-15T06:35:04.918 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:35:04.919 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2023-11-15T06:35:04.919 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:35:04.919 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2023-11-15T06:35:04.919 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2023-11-15T06:35:04.919 INFO:teuthology.orchestra.run.smithi135.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2023-11-15T06:35:04.920 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2023-11-15T06:35:04.920 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2023-11-15T06:35:04.920 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2023-11-15T06:35:04.920 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy x86_64 1.0.0-21.module_el8.5.0+771+e5d9a225 appstream 14 M 2023-11-15T06:35:04.920 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:04.921 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:35:04.921 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================================ 2023-11-15T06:35:04.921 INFO:teuthology.orchestra.run.smithi135.stdout:Install 8 Packages 2023-11-15T06:35:04.922 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:04.922 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 37 M 2023-11-15T06:35:04.923 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 210 M 2023-11-15T06:35:04.923 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:35:05.131 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:35:05.454 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:35:05.683 INFO:teuthology.orchestra.run.smithi135.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 57 MB/s | 8.6 MB 00:00 2023-11-15T06:35:05.709 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:35:06.142 INFO:teuthology.orchestra.run.smithi135.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 7.8 MB/s | 4.7 MB 00:00 2023-11-15T06:35:06.242 INFO:teuthology.orchestra.run.smithi135.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 6.8 MB/s | 4.9 MB 00:00 2023-11-15T06:35:06.267 INFO:teuthology.orchestra.run.smithi135.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2023-11-15T06:35:06.568 INFO:teuthology.orchestra.run.smithi135.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.2 MB/s | 3.7 MB 00:00 2023-11-15T06:35:06.826 INFO:teuthology.orchestra.run.smithi135.stdout:(6/8): libgfortran-8.5.0-21.el8.x86_64.rpm 1.1 MB/s | 645 kB 00:00 2023-11-15T06:35:06.985 INFO:teuthology.orchestra.run.smithi135.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 414 kB/s | 172 kB 00:00 2023-11-15T06:35:07.119 INFO:teuthology.orchestra.run.smithi135.stdout:(8/8): python3-scipy-1.0.0-21.module_el8.5.0+77 16 MB/s | 14 MB 00:00 2023-11-15T06:35:07.119 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:07.119 INFO:teuthology.orchestra.run.smithi135.stdout:Total 17 MB/s | 37 MB 00:02 2023-11-15T06:35:07.354 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:35:07.373 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:35:07.373 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:35:07.459 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:35:07.709 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:35:07.711 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:35:08.739 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T06:35:08.934 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:35:09.247 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:35:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:35:09.297 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T06:35:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T06:35:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:35:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T06:35:09.298 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:35:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T06:35:09.299 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T06:35:09.430 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:35:09.556 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2023-11-15T06:35:09.729 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:35:09.973 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T06:35:09.973 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:09.973 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:35:09.973 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T06:35:09.974 INFO:teuthology.orchestra.run.smithi123.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T06:35:09.974 INFO:teuthology.orchestra.run.smithi123.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T06:35:09.974 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T06:35:09.974 INFO:teuthology.orchestra.run.smithi123.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T06:35:09.974 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T06:35:09.975 INFO:teuthology.orchestra.run.smithi123.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T06:35:09.975 INFO:teuthology.orchestra.run.smithi123.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T06:35:09.975 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:09.975 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:10.085 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-rook 2023-11-15T06:35:10.326 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2023-11-15T06:35:10.529 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:35:10.938 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:01:48 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:11.144 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:35:11.321 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:35:12.022 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2023-11-15T06:35:12.400 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:35:12.606 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:12.608 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:12.608 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-15T06:35:12.608 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:12.609 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:35:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T06:35:12.609 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:35:12.609 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T06:35:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T06:35:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T06:35:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T06:35:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T06:35:12.610 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T06:35:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T06:35:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T06:35:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T06:35:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T06:35:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T06:35:12.611 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T06:35:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:12.612 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:35:12.612 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:12.612 INFO:teuthology.orchestra.run.smithi123.stdout:Install 13 Packages 2023-11-15T06:35:12.612 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:12.613 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 1.9 M 2023-11-15T06:35:12.613 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 24 M 2023-11-15T06:35:12.613 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:35:12.651 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:35:13.064 INFO:teuthology.orchestra.run.smithi123.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 708 kB/s | 27 kB 00:00 2023-11-15T06:35:13.206 INFO:teuthology.orchestra.run.smithi123.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 111 kB/s | 20 kB 00:00 2023-11-15T06:35:13.232 INFO:teuthology.orchestra.run.smithi123.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 129 kB/s | 27 kB 00:00 2023-11-15T06:35:13.332 INFO:teuthology.orchestra.run.smithi123.stdout:(4/13): python3-pyasn1-modules-0.3.7-6.el8.noar 878 kB/s | 110 kB 00:00 2023-11-15T06:35:13.357 INFO:teuthology.orchestra.run.smithi123.stdout:(5/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 431 kB/s | 126 kB 00:00 2023-11-15T06:35:13.741 INFO:teuthology.orchestra.run.smithi123.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 304 kB/s | 155 kB 00:00 2023-11-15T06:35:13.758 INFO:teuthology.orchestra.run.smithi123.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 100 kB/s | 43 kB 00:00 2023-11-15T06:35:14.396 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2023-11-15T06:35:14.397 INFO:teuthology.orchestra.run.smithi123.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 32 kB/s | 33 kB 00:01 2023-11-15T06:35:14.608 INFO:teuthology.orchestra.run.smithi123.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 19 kB/s | 16 kB 00:00 2023-11-15T06:35:14.975 INFO:teuthology.orchestra.run.smithi123.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 67 kB/s | 81 kB 00:01 2023-11-15T06:35:15.167 INFO:teuthology.orchestra.run.smithi123.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 115 kB/s | 64 kB 00:00 2023-11-15T06:35:15.309 INFO:teuthology.orchestra.run.smithi123.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 184 kB/s | 61 kB 00:00 2023-11-15T06:35:15.695 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 7/8 2023-11-15T06:35:15.759 INFO:teuthology.orchestra.run.smithi123.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 873 kB/s | 1.2 MB 00:01 2023-11-15T06:35:15.762 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:15.763 INFO:teuthology.orchestra.run.smithi123.stdout:Total 621 kB/s | 1.9 MB 00:03 2023-11-15T06:35:15.817 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:35:15.833 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:35:15.833 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:35:15.867 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:35:15.919 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:35:15.922 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:35:16.184 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2023-11-15T06:35:16.184 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2023-11-15T06:35:16.184 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2023-11-15T06:35:16.184 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2023-11-15T06:35:16.185 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2023-11-15T06:35:16.185 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2023-11-15T06:35:16.185 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x 6/8 2023-11-15T06:35:16.185 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2023-11-15T06:35:16.421 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:35:16.668 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T06:35:16.878 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2023-11-15T06:35:16.878 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:16.878 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:35:16.878 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2023-11-15T06:35:16.879 INFO:teuthology.orchestra.run.smithi135.stdout: libgfortran-8.5.0-21.el8.x86_64 2023-11-15T06:35:16.879 INFO:teuthology.orchestra.run.smithi135.stdout: libquadmath-8.5.0-21.el8.x86_64 2023-11-15T06:35:16.879 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-0.3.15-4.el8.x86_64 2023-11-15T06:35:16.879 INFO:teuthology.orchestra.run.smithi135.stdout: openblas-threads-0.3.15-4.el8.x86_64 2023-11-15T06:35:16.879 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2023-11-15T06:35:16.880 INFO:teuthology.orchestra.run.smithi135.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2023-11-15T06:35:16.880 INFO:teuthology.orchestra.run.smithi135.stdout: python3-scipy-1.0.0-21.module_el8.5.0+771+e5d9a225.x86_64 2023-11-15T06:35:16.880 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:16.880 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:16.881 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T06:35:16.991 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-rook 2023-11-15T06:35:17.118 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T06:35:17.314 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T06:35:17.502 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T06:35:17.736 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T06:35:17.853 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:01:52 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:18.035 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T06:35:18.253 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T06:35:19.253 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T06:35:19.509 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:19.510 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:19.511 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-11-15T06:35:19.511 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:19.511 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:35:19.511 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2023-11-15T06:35:19.511 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:35:19.512 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2023-11-15T06:35:19.512 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2023-11-15T06:35:19.512 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2023-11-15T06:35:19.512 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2023-11-15T06:35:19.512 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2023-11-15T06:35:19.513 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2023-11-15T06:35:19.513 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2023-11-15T06:35:19.513 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2023-11-15T06:35:19.513 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2023-11-15T06:35:19.513 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2023-11-15T06:35:19.513 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2023-11-15T06:35:19.514 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2023-11-15T06:35:19.514 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:19.514 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:35:19.514 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:19.514 INFO:teuthology.orchestra.run.smithi135.stdout:Install 13 Packages 2023-11-15T06:35:19.515 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:19.515 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 1.9 M 2023-11-15T06:35:19.516 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 24 M 2023-11-15T06:35:19.516 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:35:19.538 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T06:35:19.747 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T06:35:19.947 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T06:35:20.148 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:35:20.344 INFO:teuthology.orchestra.run.smithi135.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 765 kB/s | 27 kB 00:00 2023-11-15T06:35:20.450 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:35:20.451 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T06:35:20.451 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T06:35:20.451 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T06:35:20.451 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T06:35:20.451 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T06:35:20.452 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T06:35:20.452 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T06:35:20.452 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T06:35:20.452 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T06:35:20.452 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T06:35:20.453 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T06:35:20.453 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T06:35:20.555 INFO:teuthology.orchestra.run.smithi135.stdout:(2/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 108 kB/s | 27 kB 00:00 2023-11-15T06:35:20.586 INFO:teuthology.orchestra.run.smithi135.stdout:(3/13): python3-jsonpointer-1.10-11.el8.noarch. 72 kB/s | 20 kB 00:00 2023-11-15T06:35:20.629 INFO:teuthology.orchestra.run.smithi135.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 445 kB/s | 126 kB 00:00 2023-11-15T06:35:20.679 INFO:teuthology.orchestra.run.smithi135.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 1.2 MB/s | 110 kB 00:00 2023-11-15T06:35:20.829 INFO:teuthology.orchestra.run.smithi135.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 220 kB/s | 33 kB 00:00 2023-11-15T06:35:20.888 INFO:teuthology.orchestra.run.smithi135.stdout:(7/13): python3-certifi-2018.10.15-7.el8.noarch 279 kB/s | 16 kB 00:00 2023-11-15T06:35:20.963 INFO:teuthology.orchestra.run.smithi135.stdout:(8/13): python3-google-auth-1.1.1-10.el8.noarch 1.1 MB/s | 81 kB 00:00 2023-11-15T06:35:20.989 INFO:teuthology.orchestra.run.smithi135.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 119 kB/s | 43 kB 00:00 2023-11-15T06:35:21.022 INFO:teuthology.orchestra.run.smithi135.stdout:(10/13): python3-oauthlib-2.1.0-1.el8.noarch.rp 356 kB/s | 155 kB 00:00 2023-11-15T06:35:21.147 INFO:teuthology.orchestra.run.smithi135.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 404 kB/s | 64 kB 00:00 2023-11-15T06:35:21.181 INFO:teuthology.orchestra.run.smithi135.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 398 kB/s | 61 kB 00:00 2023-11-15T06:35:21.208 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T06:35:21.208 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:21.209 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:35:21.209 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T06:35:21.209 INFO:teuthology.orchestra.run.smithi123.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T06:35:21.209 INFO:teuthology.orchestra.run.smithi123.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T06:35:21.210 INFO:teuthology.orchestra.run.smithi123.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T06:35:21.210 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T06:35:21.210 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T06:35:21.210 INFO:teuthology.orchestra.run.smithi123.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T06:35:21.210 INFO:teuthology.orchestra.run.smithi123.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T06:35:21.211 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T06:35:21.211 INFO:teuthology.orchestra.run.smithi123.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T06:35:21.211 INFO:teuthology.orchestra.run.smithi123.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T06:35:21.211 INFO:teuthology.orchestra.run.smithi123.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T06:35:21.212 INFO:teuthology.orchestra.run.smithi123.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T06:35:21.212 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:21.212 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:21.340 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T06:35:21.347 INFO:teuthology.orchestra.run.smithi135.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 3.0 MB/s | 1.2 MB 00:00 2023-11-15T06:35:21.352 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:21.352 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.0 MB/s | 1.9 MB 00:01 2023-11-15T06:35:21.411 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:35:21.427 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:35:21.427 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:35:21.513 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:35:21.516 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:35:22.010 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:35:22.197 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:00 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:22.246 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2023-11-15T06:35:22.442 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2023-11-15T06:35:22.671 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2023-11-15T06:35:22.926 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2023-11-15T06:35:23.142 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2023-11-15T06:35:23.381 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2023-11-15T06:35:23.708 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2023-11-15T06:35:23.851 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:23.851 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:23.852 INFO:teuthology.orchestra.run.smithi123.stdout: Package Arch Version Repository Size 2023-11-15T06:35:23.852 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:23.852 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:35:23.852 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T06:35:23.853 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:35:23.853 INFO:teuthology.orchestra.run.smithi123.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T06:35:23.853 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T06:35:23.853 INFO:teuthology.orchestra.run.smithi123.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T06:35:23.854 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T06:35:23.854 INFO:teuthology.orchestra.run.smithi123.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T06:35:23.854 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:23.854 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:35:23.854 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:23.855 INFO:teuthology.orchestra.run.smithi123.stdout:Install 6 Packages 2023-11-15T06:35:23.855 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:23.856 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 5.8 M 2023-11-15T06:35:23.856 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 24 M 2023-11-15T06:35:23.856 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:35:23.965 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2023-11-15T06:35:24.158 INFO:teuthology.orchestra.run.smithi123.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.7 MB/s | 89 kB 00:00 2023-11-15T06:35:24.519 INFO:teuthology.orchestra.run.smithi123.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 59 kB/s | 21 kB 00:00 2023-11-15T06:35:24.752 INFO:teuthology.orchestra.run.smithi123.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 832 kB/s | 538 kB 00:00 2023-11-15T06:35:24.953 INFO:teuthology.orchestra.run.smithi123.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 820 kB/s | 355 kB 00:00 2023-11-15T06:35:25.022 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2023-11-15T06:35:25.180 INFO:teuthology.orchestra.run.smithi123.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 115 kB/s | 49 kB 00:00 2023-11-15T06:35:25.300 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2023-11-15T06:35:25.556 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2023-11-15T06:35:25.751 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2023-11-15T06:35:25.910 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:35:26.119 INFO:teuthology.orchestra.run.smithi123.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 2.4 MB/s | 4.8 MB 00:02 2023-11-15T06:35:26.120 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:26.121 INFO:teuthology.orchestra.run.smithi123.stdout:Total 2.6 MB/s | 5.8 MB 00:02 2023-11-15T06:35:26.173 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:35:26.183 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:35:26.184 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:35:26.195 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2023-11-15T06:35:26.195 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2023-11-15T06:35:26.195 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2023-11-15T06:35:26.195 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2023-11-15T06:35:26.195 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2023-11-15T06:35:26.196 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2023-11-15T06:35:26.196 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2023-11-15T06:35:26.196 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2023-11-15T06:35:26.196 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2023-11-15T06:35:26.196 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2023-11-15T06:35:26.197 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2023-11-15T06:35:26.197 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2023-11-15T06:35:26.197 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2023-11-15T06:35:26.286 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:35:26.290 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:35:26.570 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:35:26.822 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T06:35:26.962 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2023-11-15T06:35:26.962 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:26.962 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:35:26.963 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2023-11-15T06:35:26.963 INFO:teuthology.orchestra.run.smithi135.stdout: python3-cachetools-3.1.1-4.el8.noarch 2023-11-15T06:35:26.963 INFO:teuthology.orchestra.run.smithi135.stdout: python3-certifi-2018.10.15-7.el8.noarch 2023-11-15T06:35:26.963 INFO:teuthology.orchestra.run.smithi135.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2023-11-15T06:35:26.963 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpatch-1.21-2.el8.noarch 2023-11-15T06:35:26.964 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jsonpointer-1.10-11.el8.noarch 2023-11-15T06:35:26.964 INFO:teuthology.orchestra.run.smithi135.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2023-11-15T06:35:26.964 INFO:teuthology.orchestra.run.smithi135.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2023-11-15T06:35:26.964 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2023-11-15T06:35:26.965 INFO:teuthology.orchestra.run.smithi135.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2023-11-15T06:35:26.965 INFO:teuthology.orchestra.run.smithi135.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2023-11-15T06:35:26.965 INFO:teuthology.orchestra.run.smithi135.stdout: python3-rsa-4.9-2.el8.noarch 2023-11-15T06:35:26.965 INFO:teuthology.orchestra.run.smithi135.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2023-11-15T06:35:26.965 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:26.966 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:27.108 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T06:35:27.115 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-mgr-cephadm 2023-11-15T06:35:27.972 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:02 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:28.015 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T06:35:28.359 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T06:35:28.585 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T06:35:28.764 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:35:29.045 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:35:29.045 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:35:29.045 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T06:35:29.045 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T06:35:29.046 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T06:35:29.046 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T06:35:29.625 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:29.625 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:29.626 INFO:teuthology.orchestra.run.smithi135.stdout: Package Arch Version Repository Size 2023-11-15T06:35:29.626 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:29.626 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:35:29.626 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2023-11-15T06:35:29.626 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:35:29.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2023-11-15T06:35:29.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2023-11-15T06:35:29.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2023-11-15T06:35:29.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2023-11-15T06:35:29.627 INFO:teuthology.orchestra.run.smithi135.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2023-11-15T06:35:29.628 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:29.628 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:35:29.628 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:29.628 INFO:teuthology.orchestra.run.smithi135.stdout:Install 6 Packages 2023-11-15T06:35:29.628 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:29.629 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 5.8 M 2023-11-15T06:35:29.629 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 24 M 2023-11-15T06:35:29.629 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:35:29.712 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T06:35:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:29.713 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:35:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T06:35:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T06:35:29.713 INFO:teuthology.orchestra.run.smithi123.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T06:35:29.714 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:29.714 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:29.859 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install ceph-fuse 2023-11-15T06:35:30.064 INFO:teuthology.orchestra.run.smithi135.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2023-11-15T06:35:30.240 INFO:teuthology.orchestra.run.smithi135.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 121 kB/s | 21 kB 00:00 2023-11-15T06:35:30.299 INFO:teuthology.orchestra.run.smithi135.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.9 MB/s | 538 kB 00:00 2023-11-15T06:35:30.424 INFO:teuthology.orchestra.run.smithi135.stdout:(4/6): python3-execnet-1.7.1-1.el8.noarch.rpm 1.9 MB/s | 355 kB 00:00 2023-11-15T06:35:30.441 INFO:teuthology.orchestra.run.smithi135.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 345 kB/s | 49 kB 00:00 2023-11-15T06:35:30.716 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:08 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:31.125 INFO:teuthology.orchestra.run.smithi135.stdout:(6/6): python3-babel-2.5.1-7.el8.noarch.rpm 4.3 MB/s | 4.8 MB 00:01 2023-11-15T06:35:31.126 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:31.126 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.9 MB/s | 5.8 MB 00:01 2023-11-15T06:35:31.177 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:35:31.187 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:35:31.187 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:35:31.291 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:35:31.294 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:35:31.566 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:35:31.850 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2023-11-15T06:35:32.171 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2023-11-15T06:35:32.383 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:32.383 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:32.384 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:35:32.384 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:32.384 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:35:32.384 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T06:35:32.385 INFO:teuthology.orchestra.run.smithi123.stdout:Installing dependencies: 2023-11-15T06:35:32.385 INFO:teuthology.orchestra.run.smithi123.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T06:35:32.385 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:32.385 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:35:32.385 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:32.385 INFO:teuthology.orchestra.run.smithi123.stdout:Install 2 Packages 2023-11-15T06:35:32.386 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:32.386 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 968 k 2023-11-15T06:35:32.387 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 3.2 M 2023-11-15T06:35:32.387 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:35:32.636 INFO:teuthology.orchestra.run.smithi123.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-15T06:35:32.920 INFO:teuthology.orchestra.run.smithi123.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 245 kB/s | 83 kB 00:00 2023-11-15T06:35:32.920 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:32.921 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.8 MB/s | 968 kB 00:00 2023-11-15T06:35:32.931 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:35:32.952 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:35:32.953 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:35:33.040 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:35:33.044 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:35:33.129 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2023-11-15T06:35:33.233 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:35:33.477 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2023-11-15T06:35:33.507 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T06:35:33.688 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:35:33.702 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2023-11-15T06:35:33.901 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:35:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:35:34.074 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T06:35:34.172 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2023-11-15T06:35:34.172 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2023-11-15T06:35:34.172 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2023-11-15T06:35:34.172 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2023-11-15T06:35:34.173 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2023-11-15T06:35:34.173 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2023-11-15T06:35:34.723 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T06:35:34.723 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:34.724 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:35:34.724 INFO:teuthology.orchestra.run.smithi123.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T06:35:34.724 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:34.724 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:34.841 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2023-11-15T06:35:34.842 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:34.842 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:35:34.842 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2023-11-15T06:35:34.843 INFO:teuthology.orchestra.run.smithi135.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2023-11-15T06:35:34.843 INFO:teuthology.orchestra.run.smithi135.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2023-11-15T06:35:34.843 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:34.843 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:34.977 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install ceph-fuse 2023-11-15T06:35:35.834 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:10 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:36.237 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados-devel 2023-11-15T06:35:37.111 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:37.498 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:37.499 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:37.499 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:35:37.499 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:37.500 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:35:37.500 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2023-11-15T06:35:37.500 INFO:teuthology.orchestra.run.smithi135.stdout:Installing dependencies: 2023-11-15T06:35:37.501 INFO:teuthology.orchestra.run.smithi135.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2023-11-15T06:35:37.501 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:37.501 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:35:37.502 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:37.502 INFO:teuthology.orchestra.run.smithi135.stdout:Install 2 Packages 2023-11-15T06:35:37.503 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:37.504 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 968 k 2023-11-15T06:35:37.505 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 3.2 M 2023-11-15T06:35:37.505 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:35:37.594 INFO:teuthology.orchestra.run.smithi135.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 16 MB/s | 885 kB 00:00 2023-11-15T06:35:37.869 INFO:teuthology.orchestra.run.smithi135.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 251 kB/s | 83 kB 00:00 2023-11-15T06:35:37.870 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:37.870 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.6 MB/s | 968 kB 00:00 2023-11-15T06:35:37.880 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:35:37.902 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:35:37.903 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:35:37.986 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:35:37.989 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:35:38.153 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:35:38.442 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2023-11-15T06:35:38.608 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:35:38.823 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:38.824 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:35:38.825 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:38.825 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:35:38.825 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T06:35:38.826 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:38.826 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:35:38.826 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:38.826 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-11-15T06:35:38.827 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:38.827 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 131 k 2023-11-15T06:35:38.827 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 439 k 2023-11-15T06:35:38.827 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:35:38.867 INFO:teuthology.orchestra.run.smithi123.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 131 kB 00:00 2023-11-15T06:35:38.868 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:38.868 INFO:teuthology.orchestra.run.smithi123.stdout:Total 3.0 MB/s | 131 kB 00:00 2023-11-15T06:35:38.868 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:35:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:35:38.873 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:35:38.922 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:35:38.925 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:35:39.007 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2023-11-15T06:35:39.007 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2023-11-15T06:35:39.035 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:35:39.253 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:39.543 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:39.712 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2023-11-15T06:35:39.712 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:39.713 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:35:39.713 INFO:teuthology.orchestra.run.smithi135.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2023-11-15T06:35:39.713 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:39.713 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:39.927 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados-devel 2023-11-15T06:35:40.195 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:40.195 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:40.196 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:35:40.196 INFO:teuthology.orchestra.run.smithi123.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:35:40.196 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:40.196 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:40.429 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs2 2023-11-15T06:35:40.794 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:15 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:41.295 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:19 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:42.453 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:42.454 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:42.454 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:35:42.454 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:42.454 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:35:42.454 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2023-11-15T06:35:42.455 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:42.455 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:35:42.455 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:42.455 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-11-15T06:35:42.455 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:42.456 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 131 k 2023-11-15T06:35:42.456 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 439 k 2023-11-15T06:35:42.456 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:35:42.500 INFO:teuthology.orchestra.run.smithi135.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.9 MB/s | 131 kB 00:00 2023-11-15T06:35:42.500 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:42.500 INFO:teuthology.orchestra.run.smithi135.stdout:Total 2.8 MB/s | 131 kB 00:00 2023-11-15T06:35:42.501 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:35:42.506 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:35:42.506 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:35:42.555 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:35:42.558 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:35:42.689 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:35:42.742 INFO:teuthology.orchestra.run.smithi123.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:42.940 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:42.956 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:42.957 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:35:42.957 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:43.059 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install libcephfs-devel 2023-11-15T06:35:43.218 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:43.874 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:43.875 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:43.875 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:35:43.875 INFO:teuthology.orchestra.run.smithi135.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:35:43.875 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:43.876 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:43.922 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:21 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:44.128 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs2 2023-11-15T06:35:44.992 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:19 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:45.581 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:45.582 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:45.582 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:35:45.582 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:45.583 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:35:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T06:35:45.583 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:45.584 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:35:45.584 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:35:45.584 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-11-15T06:35:45.584 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:45.585 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 24 k 2023-11-15T06:35:45.585 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 94 k 2023-11-15T06:35:45.585 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:35:45.646 INFO:teuthology.orchestra.run.smithi123.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 384 kB/s | 24 kB 00:00 2023-11-15T06:35:45.647 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:45.647 INFO:teuthology.orchestra.run.smithi123.stdout:Total 377 kB/s | 24 kB 00:00 2023-11-15T06:35:45.648 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:35:45.650 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:35:45.651 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:35:45.653 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:35:45.656 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:35:45.709 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:35:45.893 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:46.160 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:46.425 INFO:teuthology.orchestra.run.smithi135.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:46.642 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:46.644 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:35:46.644 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:46.782 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:46.783 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:35:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:35:46.783 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:35:46.783 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:46.910 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librados2 2023-11-15T06:35:47.410 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install libcephfs-devel 2023-11-15T06:35:47.770 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:25 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:48.275 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:23 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:49.223 INFO:teuthology.orchestra.run.smithi123.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:49.438 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:49.439 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:35:49.439 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:49.556 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install librbd1 2023-11-15T06:35:49.938 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:49.939 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:49.940 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:35:49.940 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:49.940 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:35:49.940 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2023-11-15T06:35:49.941 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:49.941 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:35:49.941 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:35:49.941 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-11-15T06:35:49.941 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:49.942 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 24 k 2023-11-15T06:35:49.942 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 94 k 2023-11-15T06:35:49.942 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:35:50.007 INFO:teuthology.orchestra.run.smithi135.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 365 kB/s | 24 kB 00:00 2023-11-15T06:35:50.008 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:35:50.008 INFO:teuthology.orchestra.run.smithi135.stdout:Total 359 kB/s | 24 kB 00:00 2023-11-15T06:35:50.008 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:35:50.011 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:35:50.011 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:35:50.013 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:35:50.017 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:35:50.063 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:35:50.264 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:50.449 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:28 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:50.521 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:51.159 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:35:51.159 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:51.159 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:35:51.160 INFO:teuthology.orchestra.run.smithi135.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2023-11-15T06:35:51.160 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:35:51.160 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:51.282 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librados2 2023-11-15T06:35:51.893 INFO:teuthology.orchestra.run.smithi123.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:52.108 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:52.110 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:35:52.110 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:52.143 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:27 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:52.210 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rados 2023-11-15T06:35:53.073 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:31 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:53.581 INFO:teuthology.orchestra.run.smithi135.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:53.799 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:53.799 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:35:53.799 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:53.900 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install librbd1 2023-11-15T06:35:54.527 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:54.746 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:54.748 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:35:54.749 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:54.757 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:29 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:54.913 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rgw 2023-11-15T06:35:55.776 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:33 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:56.200 INFO:teuthology.orchestra.run.smithi135.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:56.416 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:56.417 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:35:56.417 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:56.524 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rados 2023-11-15T06:35:57.222 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:57.394 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:32 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:35:57.440 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:35:57.441 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:35:57.442 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:35:57.555 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-cephfs 2023-11-15T06:35:58.415 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:36 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:35:58.862 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:35:59.106 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:35:59.107 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:35:59.107 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:35:59.240 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rgw 2023-11-15T06:35:59.842 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:36:00.058 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:36:00.059 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:36:00.059 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:36:00.093 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:35 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:36:00.165 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install python3-rbd 2023-11-15T06:36:01.029 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:39 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:36:01.542 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:36:01.760 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:36:01.761 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:36:01.761 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:36:01.861 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-cephfs 2023-11-15T06:36:02.481 INFO:teuthology.orchestra.run.smithi123.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:36:02.698 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:36:02.698 INFO:teuthology.orchestra.run.smithi123.stdout:Nothing to do. 2023-11-15T06:36:02.699 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:36:02.719 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:37 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:36:02.806 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-fuse 2023-11-15T06:36:03.666 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:41 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:36:04.160 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:36:04.390 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:36:04.391 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:36:04.392 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:36:04.492 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install python3-rbd 2023-11-15T06:36:05.342 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:36:05.344 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:05.344 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:36:05.345 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:05.346 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:36:05.346 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T06:36:05.346 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:05.347 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:36:05.347 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:05.347 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-11-15T06:36:05.348 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:05.348 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 89 k 2023-11-15T06:36:05.348 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 298 k 2023-11-15T06:36:05.349 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:36:05.353 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:40 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:36:05.394 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.8 MB/s | 89 kB 00:00 2023-11-15T06:36:05.395 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:36:05.395 INFO:teuthology.orchestra.run.smithi123.stdout:Total 1.7 MB/s | 89 kB 00:00 2023-11-15T06:36:05.396 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:36:05.404 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:36:05.405 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:36:05.453 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:36:05.456 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:36:05.566 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:36:05.773 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:06.066 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:06.741 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:06.741 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:06.742 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:36:06.742 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T06:36:06.742 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:06.742 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:36:06.808 INFO:teuthology.orchestra.run.smithi135.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2023-11-15T06:36:06.924 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-mirror 2023-11-15T06:36:07.021 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:36:07.022 INFO:teuthology.orchestra.run.smithi135.stdout:Nothing to do. 2023-11-15T06:36:07.022 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:36:07.115 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-fuse 2023-11-15T06:36:07.787 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:45 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:36:07.976 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:42 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:36:09.454 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:36:09.455 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:09.455 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:36:09.455 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:09.456 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:36:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:36:09.456 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:09.456 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:36:09.456 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:09.457 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-11-15T06:36:09.457 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:09.457 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 3.7 M 2023-11-15T06:36:09.457 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 18 M 2023-11-15T06:36:09.457 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:36:09.545 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 41 MB/s | 3.7 MB 00:00 2023-11-15T06:36:09.546 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:36:09.546 INFO:teuthology.orchestra.run.smithi123.stdout:Total 41 MB/s | 3.7 MB 00:00 2023-11-15T06:36:09.547 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:36:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:36:09.558 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:36:09.622 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:36:09.625 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:36:09.633 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:36:09.633 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:09.634 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:36:09.634 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:09.634 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:36:09.634 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2023-11-15T06:36:09.634 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:09.635 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:36:09.635 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:09.635 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-11-15T06:36:09.635 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:09.636 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 89 k 2023-11-15T06:36:09.636 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 298 k 2023-11-15T06:36:09.636 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:36:09.697 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.4 MB/s | 89 kB 00:00 2023-11-15T06:36:09.698 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:36:09.698 INFO:teuthology.orchestra.run.smithi135.stdout:Total 1.4 MB/s | 89 kB 00:00 2023-11-15T06:36:09.698 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:36:09.708 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:36:09.708 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:36:09.756 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:36:09.759 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:36:09.875 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:36:10.045 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:10.092 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:36:10.317 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:10.344 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:10.733 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:11.024 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:11.024 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:11.025 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:36:11.025 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2023-11-15T06:36:11.025 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:11.025 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:36:11.248 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-mirror 2023-11-15T06:36:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:11.404 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:11.405 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:36:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T06:36:11.405 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:11.405 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:36:11.632 DEBUG:teuthology.orchestra.run.smithi123:> sudo yum -y install rbd-nbd 2023-11-15T06:36:12.110 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:47 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:36:12.501 INFO:teuthology.orchestra.run.smithi123.stdout:Last metadata expiration check: 0:02:50 ago on Wed 15 Nov 2023 06:33:22 AM UTC. 2023-11-15T06:36:13.765 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:36:13.766 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:13.766 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:36:13.766 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:13.766 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:36:13.766 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2023-11-15T06:36:13.767 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:13.767 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:36:13.767 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:13.767 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-11-15T06:36:13.767 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:13.768 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 3.7 M 2023-11-15T06:36:13.768 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 18 M 2023-11-15T06:36:13.768 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:36:13.850 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 44 MB/s | 3.7 MB 00:00 2023-11-15T06:36:13.851 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:36:13.851 INFO:teuthology.orchestra.run.smithi135.stdout:Total 43 MB/s | 3.7 MB 00:00 2023-11-15T06:36:13.852 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:36:13.863 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:36:13.863 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:36:13.925 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:36:13.928 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:36:14.159 INFO:teuthology.orchestra.run.smithi123.stdout:Dependencies resolved. 2023-11-15T06:36:14.160 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:14.161 INFO:teuthology.orchestra.run.smithi123.stdout: Package Architecture Version Repository Size 2023-11-15T06:36:14.161 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:14.161 INFO:teuthology.orchestra.run.smithi123.stdout:Installing: 2023-11-15T06:36:14.161 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T06:36:14.161 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:14.162 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction Summary 2023-11-15T06:36:14.162 INFO:teuthology.orchestra.run.smithi123.stdout:================================================================================ 2023-11-15T06:36:14.162 INFO:teuthology.orchestra.run.smithi123.stdout:Install 1 Package 2023-11-15T06:36:14.162 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:14.163 INFO:teuthology.orchestra.run.smithi123.stdout:Total download size: 173 k 2023-11-15T06:36:14.163 INFO:teuthology.orchestra.run.smithi123.stdout:Installed size: 649 k 2023-11-15T06:36:14.163 INFO:teuthology.orchestra.run.smithi123.stdout:Downloading Packages: 2023-11-15T06:36:14.206 INFO:teuthology.orchestra.run.smithi123.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.8 MB/s | 173 kB 00:00 2023-11-15T06:36:14.207 INFO:teuthology.orchestra.run.smithi123.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:36:14.207 INFO:teuthology.orchestra.run.smithi123.stdout:Total 3.7 MB/s | 173 kB 00:00 2023-11-15T06:36:14.207 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction check 2023-11-15T06:36:14.217 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction check succeeded. 2023-11-15T06:36:14.217 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction test 2023-11-15T06:36:14.266 INFO:teuthology.orchestra.run.smithi123.stdout:Transaction test succeeded. 2023-11-15T06:36:14.269 INFO:teuthology.orchestra.run.smithi123.stdout:Running transaction 2023-11-15T06:36:14.383 INFO:teuthology.orchestra.run.smithi123.stdout: Preparing : 1/1 2023-11-15T06:36:14.400 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:36:14.563 INFO:teuthology.orchestra.run.smithi123.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:14.695 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:14.830 INFO:teuthology.orchestra.run.smithi123.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:15.121 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:15.489 INFO:teuthology.orchestra.run.smithi123.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:15.490 INFO:teuthology.orchestra.run.smithi123.stdout:Installed: 2023-11-15T06:36:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:36:15.490 INFO:teuthology.orchestra.run.smithi123.stdout: 2023-11-15T06:36:15.491 INFO:teuthology.orchestra.run.smithi123.stdout:Complete! 2023-11-15T06:36:15.719 DEBUG:teuthology.parallel:result is None 2023-11-15T06:36:15.813 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:15.813 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:15.813 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:36:15.813 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2023-11-15T06:36:15.814 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:15.814 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:36:16.033 DEBUG:teuthology.orchestra.run.smithi135:> sudo yum -y install rbd-nbd 2023-11-15T06:36:16.894 INFO:teuthology.orchestra.run.smithi135.stdout:Last metadata expiration check: 0:02:51 ago on Wed 15 Nov 2023 06:33:25 AM UTC. 2023-11-15T06:36:18.571 INFO:teuthology.orchestra.run.smithi135.stdout:Dependencies resolved. 2023-11-15T06:36:18.572 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:18.572 INFO:teuthology.orchestra.run.smithi135.stdout: Package Architecture Version Repository Size 2023-11-15T06:36:18.573 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:18.573 INFO:teuthology.orchestra.run.smithi135.stdout:Installing: 2023-11-15T06:36:18.573 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2023-11-15T06:36:18.573 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:18.573 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction Summary 2023-11-15T06:36:18.574 INFO:teuthology.orchestra.run.smithi135.stdout:================================================================================ 2023-11-15T06:36:18.574 INFO:teuthology.orchestra.run.smithi135.stdout:Install 1 Package 2023-11-15T06:36:18.574 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:18.574 INFO:teuthology.orchestra.run.smithi135.stdout:Total download size: 173 k 2023-11-15T06:36:18.575 INFO:teuthology.orchestra.run.smithi135.stdout:Installed size: 649 k 2023-11-15T06:36:18.575 INFO:teuthology.orchestra.run.smithi135.stdout:Downloading Packages: 2023-11-15T06:36:18.621 INFO:teuthology.orchestra.run.smithi135.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.5 MB/s | 173 kB 00:00 2023-11-15T06:36:18.622 INFO:teuthology.orchestra.run.smithi135.stdout:-------------------------------------------------------------------------------- 2023-11-15T06:36:18.622 INFO:teuthology.orchestra.run.smithi135.stdout:Total 3.5 MB/s | 173 kB 00:00 2023-11-15T06:36:18.622 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction check 2023-11-15T06:36:18.632 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction check succeeded. 2023-11-15T06:36:18.632 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction test 2023-11-15T06:36:18.682 INFO:teuthology.orchestra.run.smithi135.stdout:Transaction test succeeded. 2023-11-15T06:36:18.685 INFO:teuthology.orchestra.run.smithi135.stdout:Running transaction 2023-11-15T06:36:18.812 INFO:teuthology.orchestra.run.smithi135.stdout: Preparing : 1/1 2023-11-15T06:36:19.001 INFO:teuthology.orchestra.run.smithi135.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:19.284 INFO:teuthology.orchestra.run.smithi135.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:19.976 INFO:teuthology.orchestra.run.smithi135.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2023-11-15T06:36:19.976 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:19.976 INFO:teuthology.orchestra.run.smithi135.stdout:Installed: 2023-11-15T06:36:19.976 INFO:teuthology.orchestra.run.smithi135.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2023-11-15T06:36:19.977 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:36:19.977 INFO:teuthology.orchestra.run.smithi135.stdout:Complete! 2023-11-15T06:36:20.174 DEBUG:teuthology.parallel:result is None 2023-11-15T06:36:20.175 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:36:20.175 INFO:teuthology.packaging:ref: None 2023-11-15T06:36:20.175 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:36:20.176 INFO:teuthology.packaging:branch: None 2023-11-15T06:36:20.176 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:36:20.382 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:36:20.383 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:36:20.603 DEBUG:teuthology.orchestra.run.smithi123:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T06:36:20.631 INFO:teuthology.orchestra.run.smithi123.stdout:16.2.4-0.el8 2023-11-15T06:36:20.632 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T06:36:20.632 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T06:36:20.633 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2023-11-15T06:36:20.633 INFO:teuthology.packaging:ref: None 2023-11-15T06:36:20.634 INFO:teuthology.packaging:tag: v16.2.4 2023-11-15T06:36:20.634 INFO:teuthology.packaging:branch: None 2023-11-15T06:36:20.634 INFO:teuthology.packaging:sha1: 620f25c1794f62207424fc9319427943fb8d1fae 2023-11-15T06:36:20.825 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:36:20.827 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2023-11-15T06:36:20.970 DEBUG:teuthology.orchestra.run.smithi135:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2023-11-15T06:36:20.995 INFO:teuthology.orchestra.run.smithi135.stdout:16.2.4-0.el8 2023-11-15T06:36:20.995 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2023-11-15T06:36:20.995 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2023-11-15T06:36:20.996 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2023-11-15T06:36:20.996 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:36:20.997 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T06:36:21.031 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:36:21.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2023-11-15T06:36:21.064 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2023-11-15T06:36:21.065 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:36:21.065 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T06:36:21.104 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T06:36:21.175 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:36:21.176 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/daemon-helper 2023-11-15T06:36:21.204 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/daemon-helper 2023-11-15T06:36:21.272 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2023-11-15T06:36:21.272 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:36:21.273 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T06:36:21.303 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T06:36:21.371 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:36:21.372 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/adjust-ulimits 2023-11-15T06:36:21.399 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2023-11-15T06:36:21.466 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2023-11-15T06:36:21.466 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:36:21.467 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T06:36:21.498 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T06:36:21.567 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:36:21.567 DEBUG:teuthology.orchestra.run.smithi135:> sudo dd of=/usr/bin/stdin-killer 2023-11-15T06:36:21.595 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod a=rx -- /usr/bin/stdin-killer 2023-11-15T06:36:21.662 INFO:teuthology.run_tasks:Running task print... 2023-11-15T06:36:21.674 INFO:teuthology.task.print:**** done install task... 2023-11-15T06:36:21.675 INFO:teuthology.run_tasks:Running task cephadm... 2023-11-15T06:36:21.777 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': '620f25c1794f62207424fc9319427943fb8d1fae'} 2023-11-15T06:36:21.778 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2023-11-15T06:36:21.779 INFO:tasks.cephadm:Cluster fsid is 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:21.779 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2023-11-15T06:36:21.779 INFO:tasks.cephadm:No mon roles; fabricating mons 2023-11-15T06:36:21.779 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi123': '172.21.15.123', 'mon.smithi135': '172.21.15.135'} 2023-11-15T06:36:21.780 INFO:tasks.cephadm:Normalizing hostnames... 2023-11-15T06:36:21.780 DEBUG:teuthology.orchestra.run.smithi123:> sudo hostname $(hostname -s) 2023-11-15T06:36:21.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo hostname $(hostname -s) 2023-11-15T06:36:21.839 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2023-11-15T06:36:21.840 DEBUG:teuthology.orchestra.run.smithi123:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T06:36:22.068 INFO:teuthology.orchestra.run.smithi123.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 06:36 /home/ubuntu/cephtest/cephadm 2023-11-15T06:36:22.069 DEBUG:teuthology.orchestra.run.smithi135:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2023-11-15T06:36:22.206 INFO:teuthology.orchestra.run.smithi135.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Nov 15 06:36 /home/ubuntu/cephtest/cephadm 2023-11-15T06:36:22.206 DEBUG:teuthology.orchestra.run.smithi123:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T06:36:22.229 DEBUG:teuthology.orchestra.run.smithi135:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2023-11-15T06:36:22.258 DEBUG:teuthology.orchestra.run.smithi123:> sudo mkdir -p /etc/ceph 2023-11-15T06:36:22.300 DEBUG:teuthology.orchestra.run.smithi135:> sudo mkdir -p /etc/ceph 2023-11-15T06:36:22.327 DEBUG:teuthology.orchestra.run.smithi123:> sudo chmod 777 /etc/ceph 2023-11-15T06:36:22.369 DEBUG:teuthology.orchestra.run.smithi135:> sudo chmod 777 /etc/ceph 2023-11-15T06:36:22.396 INFO:tasks.cephadm:Writing seed config... 2023-11-15T06:36:22.397 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2023-11-15T06:36:22.397 INFO:tasks.cephadm: override: [client] debug client = 20 2023-11-15T06:36:22.397 INFO:tasks.cephadm: override: [client] debug ms = 1 2023-11-15T06:36:22.398 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2023-11-15T06:36:22.398 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2023-11-15T06:36:22.398 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2023-11-15T06:36:22.398 INFO:tasks.cephadm: override: [mds] debug mds = 20 2023-11-15T06:36:22.399 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2023-11-15T06:36:22.399 INFO:tasks.cephadm: override: [mds] debug ms = 1 2023-11-15T06:36:22.399 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2023-11-15T06:36:22.400 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2023-11-15T06:36:22.400 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2023-11-15T06:36:22.400 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2023-11-15T06:36:22.400 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2023-11-15T06:36:22.401 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2023-11-15T06:36:22.401 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2023-11-15T06:36:22.401 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2023-11-15T06:36:22.401 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2023-11-15T06:36:22.402 INFO:tasks.cephadm: override: [mon] debug mon = 20 2023-11-15T06:36:22.402 INFO:tasks.cephadm: override: [mon] debug ms = 1 2023-11-15T06:36:22.402 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2023-11-15T06:36:22.402 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2023-11-15T06:36:22.403 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2023-11-15T06:36:22.403 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2023-11-15T06:36:22.403 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2023-11-15T06:36:22.403 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2023-11-15T06:36:22.404 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2023-11-15T06:36:22.405 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2023-11-15T06:36:22.405 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2023-11-15T06:36:22.405 INFO:tasks.cephadm: override: [osd] debug ms = 1 2023-11-15T06:36:22.405 INFO:tasks.cephadm: override: [osd] debug osd = 20 2023-11-15T06:36:22.406 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2023-11-15T06:36:22.406 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2023-11-15T06:36:22.406 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2023-11-15T06:36:22.407 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2023-11-15T06:36:22.407 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2023-11-15T06:36:22.407 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2023-11-15T06:36:22.407 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2023-11-15T06:36:22.408 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:36:22.409 DEBUG:teuthology.orchestra.run.smithi123:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2023-11-15T06:36:22.428 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2023-11-15T06:36:22.429 DEBUG:teuthology.orchestra.run.smithi123:mon.smithi123> sudo journalctl -f -n 0 -u ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123.service 2023-11-15T06:36:22.472 INFO:tasks.cephadm:Bootstrapping... 2023-11-15T06:36:22.472 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.123 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2023-11-15T06:36:22.499 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:-- Logs begin at Wed 2023-11-15 06:24:38 UTC. -- 2023-11-15T06:36:22.737 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T06:36:23.586 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: 4.6.1 2023-11-15T06:36:23.596 INFO:teuthology.orchestra.run.smithi123.stderr:Verifying podman|docker is present... 2023-11-15T06:36:23.596 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T06:36:23.619 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: 4.6.1 2023-11-15T06:36:23.630 INFO:teuthology.orchestra.run.smithi123.stderr:Verifying lvm2 is present... 2023-11-15T06:36:23.630 INFO:teuthology.orchestra.run.smithi123.stderr:Verifying time synchronization is in place... 2023-11-15T06:36:23.630 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T06:36:23.637 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T06:36:23.638 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active chrony.service 2023-11-15T06:36:23.643 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: inactive 2023-11-15T06:36:23.644 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T06:36:23.650 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: enabled 2023-11-15T06:36:23.651 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T06:36:23.656 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: active 2023-11-15T06:36:23.657 INFO:teuthology.orchestra.run.smithi123.stderr:Unit chronyd.service is enabled and running 2023-11-15T06:36:23.657 INFO:teuthology.orchestra.run.smithi123.stderr:Repeating the final host check... 2023-11-15T06:36:23.658 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2023-11-15T06:36:23.678 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: 4.6.1 2023-11-15T06:36:23.691 INFO:teuthology.orchestra.run.smithi123.stderr:podman|docker (/bin/podman) is present 2023-11-15T06:36:23.691 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl is present 2023-11-15T06:36:23.691 INFO:teuthology.orchestra.run.smithi123.stderr:lvcreate is present 2023-11-15T06:36:23.692 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled chrony.service 2023-11-15T06:36:23.697 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2023-11-15T06:36:23.697 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active chrony.service 2023-11-15T06:36:23.703 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: inactive 2023-11-15T06:36:23.704 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled chronyd.service 2023-11-15T06:36:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: enabled 2023-11-15T06:36:23.709 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active chronyd.service 2023-11-15T06:36:23.715 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: active 2023-11-15T06:36:23.715 INFO:teuthology.orchestra.run.smithi123.stderr:Unit chronyd.service is enabled and running 2023-11-15T06:36:23.716 INFO:teuthology.orchestra.run.smithi123.stderr:Host looks OK 2023-11-15T06:36:23.716 INFO:teuthology.orchestra.run.smithi123.stderr:Cluster fsid: 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:Acquiring lock 139921009326568 on /run/cephadm/4a6dbac0-8381-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:36:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:Lock 139921009326568 acquired on /run/cephadm/4a6dbac0-8381-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:36:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:Verifying IP 172.21.15.123 port 3300 ... 2023-11-15T06:36:23.717 INFO:teuthology.orchestra.run.smithi123.stderr:Verifying IP 172.21.15.123 port 6789 ... 2023-11-15T06:36:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:Base mon IP is 172.21.15.123, final addrv is [v2:172.21.15.123:3300,v1:172.21.15.123:6789] 2023-11-15T06:36:23.718 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /sbin/ip route ls 2023-11-15T06:36:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.123 metric 100 2023-11-15T06:36:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.123 metric 100 2023-11-15T06:36:23.720 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /sbin/ip -6 route ls 2023-11-15T06:36:23.722 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2023-11-15T06:36:23.722 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2023-11-15T06:36:23.722 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1792sec hoplimit 64 pref medium 2023-11-15T06:36:23.723 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /sbin/ip -6 addr ls 2023-11-15T06:36:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2023-11-15T06:36:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: inet6 ::1/128 scope host 2023-11-15T06:36:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T06:36:23.725 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2023-11-15T06:36:23.726 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:d3f1/64 scope link 2023-11-15T06:36:23.726 INFO:teuthology.orchestra.run.smithi123.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2023-11-15T06:36:23.726 INFO:teuthology.orchestra.run.smithi123.stderr:Mon IP 172.21.15.123 is in CIDR network 172.21.0.0/20 2023-11-15T06:36:23.727 INFO:teuthology.orchestra.run.smithi123.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2023-11-15T06:36:23.727 INFO:teuthology.orchestra.run.smithi123.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2023-11-15T06:36:23.727 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2023-11-15T06:36:23.817 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2023-11-15T06:36:30.427 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: Getting image source signatures 2023-11-15T06:36:30.432 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2023-11-15T06:36:30.433 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2023-11-15T06:36:42.993 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T06:36:43.051 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: Writing manifest to image destination 2023-11-15T06:36:43.357 INFO:teuthology.orchestra.run.smithi123.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2023-11-15T06:36:43.370 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2023-11-15T06:36:43.942 INFO:teuthology.orchestra.run.smithi123.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T06:36:44.347 INFO:teuthology.orchestra.run.smithi123.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2023-11-15T06:36:44.347 INFO:teuthology.orchestra.run.smithi123.stderr:Extracting ceph user uid/gid from container image... 2023-11-15T06:36:44.347 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2023-11-15T06:36:44.760 INFO:teuthology.orchestra.run.smithi123.stderr:stat: 167 167 2023-11-15T06:36:45.089 INFO:teuthology.orchestra.run.smithi123.stderr:Creating initial keys... 2023-11-15T06:36:45.089 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T06:36:45.390 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-authtool: AQD9ZlRl1flXFhAARZXSVkjwzvUczhFK7LjhIQ== 2023-11-15T06:36:45.723 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T06:36:45.998 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-authtool: AQD9ZlRlSmbXOhAAUjTAbOz47Y8ywnBXGeMhCw== 2023-11-15T06:36:46.341 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2023-11-15T06:36:46.614 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-authtool: AQD+ZlRl1E6MIxAAU7ChCF0qId7w4JTLQ4SqMA== 2023-11-15T06:36:46.957 INFO:teuthology.orchestra.run.smithi123.stderr:Creating initial monmap... 2023-11-15T06:36:46.957 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmptn_hse_d:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 --addv smithi123 [v2:172.21.15.123:3300,v1:172.21.15.123:6789] /tmp/monmap 2023-11-15T06:36:47.264 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T06:36:47.265 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:47.265 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T06:36:47.604 INFO:teuthology.orchestra.run.smithi123.stderr:monmaptool for smithi123 [v2:172.21.15.123:3300,v1:172.21.15.123:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2023-11-15T06:36:47.604 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/monmaptool: set fsid to 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:47.604 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2023-11-15T06:36:47.605 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:36:47.605 INFO:teuthology.orchestra.run.smithi123.stderr:Creating mon... 2023-11-15T06:36:47.605 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/mon.smithi123:/var/lib/ceph/mon/ceph-smithi123:z -v /tmp/ceph-tmpm_47g96v:/tmp/keyring:z -v /tmp/ceph-tmptn_hse_d:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi123 --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2023-11-15T06:36:47.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.855+0000 7f626667c700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T06:36:47.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.855+0000 7f626667c700 1 imported monmap: 2023-11-15T06:36:47.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: epoch 0 2023-11-15T06:36:47.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:47.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: last_changed 2023-11-15T06:36:47.245818+0000 2023-11-15T06:36:47.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: created 2023-11-15T06:36:47.245818+0000 2023-11-15T06:36:47.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2023-11-15T06:36:47.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: election_strategy: 1 2023-11-15T06:36:47.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon.smithi123 2023-11-15T06:36:47.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.855+0000 7f626667c700 0 /usr/bin/ceph-mon: set fsid to 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:47.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T06:36:47.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.894 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T06:36:47.894 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Compile date May 13 2021 2023-11-15T06:36:47.894 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: DB SUMMARY 2023-11-15T06:36:47.895 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.895 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi123/store.db dir, Total Num: 0, files: 2023-11-15T06:36:47.896 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.896 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi123/store.db: 2023-11-15T06:36:47.896 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.896 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T06:36:47.896 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.create_if_missing: 1 2023-11-15T06:36:47.897 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T06:36:47.897 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.env: 0x55ba162321c0 2023-11-15T06:36:47.897 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.fs: Posix File System 2023-11-15T06:36:47.897 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.info_log: 0x55ba17eceec0 2023-11-15T06:36:47.897 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T06:36:47.898 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.statistics: (nil) 2023-11-15T06:36:47.898 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.use_fsync: 0 2023-11-15T06:36:47.898 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T06:36:47.898 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T06:36:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T06:36:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T06:36:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T06:36:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T06:36:47.899 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T06:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T06:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T06:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T06:36:47.900 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T06:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.db_log_dir: 2023-11-15T06:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi123/store.db 2023-11-15T06:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T06:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T06:36:47.901 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T06:36:47.902 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T06:36:47.902 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T06:36:47.902 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T06:36:47.902 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T06:36:47.902 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T06:36:47.903 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T06:36:47.903 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T06:36:47.903 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.write_buffer_manager: 0x55ba17ed5530 2023-11-15T06:36:47.903 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T06:36:47.903 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T06:36:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T06:36:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T06:36:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T06:36:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T06:36:47.904 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T06:36:47.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T06:36:47.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T06:36:47.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.unordered_write: 0 2023-11-15T06:36:47.906 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T06:36:47.906 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T06:36:47.906 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T06:36:47.906 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T06:36:47.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.row_cache: None 2023-11-15T06:36:47.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.wal_filter: None 2023-11-15T06:36:47.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T06:36:47.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T06:36:47.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T06:36:47.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T06:36:47.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T06:36:47.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T06:36:47.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T06:36:47.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T06:36:47.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T06:36:47.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T06:36:47.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T06:36:47.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T06:36:47.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T06:36:47.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T06:36:47.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T06:36:47.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T06:36:47.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T06:36:47.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T06:36:47.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T06:36:47.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T06:36:47.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T06:36:47.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.max_open_files: -1 2023-11-15T06:36:47.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T06:36:47.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T06:36:47.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T06:36:47.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T06:36:47.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Compression algorithms supported: 2023-11-15T06:36:47.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T06:36:47.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kZSTD supported: 0 2023-11-15T06:36:47.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T06:36:47.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T06:36:47.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T06:36:47.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T06:36:47.914 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T06:36:47.914 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T06:36:47.914 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T06:36:47.914 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2023-11-15T06:36:47.914 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.856+0000 7f626667c700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000001 2023-11-15T06:36:47.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T06:36:47.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T06:36:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.merge_operator: 2023-11-15T06:36:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_filter: None 2023-11-15T06:36:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T06:36:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T06:36:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T06:36:47.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ba17de6120) 2023-11-15T06:36:47.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2023-11-15T06:36:47.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T06:36:47.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T06:36:47.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2023-11-15T06:36:47.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: index_type: 0 2023-11-15T06:36:47.918 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2023-11-15T06:36:47.918 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: index_shortening: 1 2023-11-15T06:36:47.918 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2023-11-15T06:36:47.918 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2023-11-15T06:36:47.918 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: checksum: 1 2023-11-15T06:36:47.919 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2023-11-15T06:36:47.919 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_cache: 0x55ba17e1cd10 2023-11-15T06:36:47.919 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2023-11-15T06:36:47.919 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_cache_options: 2023-11-15T06:36:47.919 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: capacity : 536870912 2023-11-15T06:36:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2023-11-15T06:36:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2023-11-15T06:36:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2023-11-15T06:36:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2023-11-15T06:36:47.920 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2023-11-15T06:36:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_size: 4096 2023-11-15T06:36:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2023-11-15T06:36:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2023-11-15T06:36:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2023-11-15T06:36:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2023-11-15T06:36:47.921 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: partition_filters: 0 2023-11-15T06:36:47.922 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2023-11-15T06:36:47.922 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T06:36:47.922 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2023-11-15T06:36:47.922 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: verify_compression: 0 2023-11-15T06:36:47.922 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2023-11-15T06:36:47.923 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: format_version: 2 2023-11-15T06:36:47.923 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2023-11-15T06:36:47.923 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: block_align: 0 2023-11-15T06:36:47.923 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.923 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T06:36:47.924 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T06:36:47.924 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compression: NoCompression 2023-11-15T06:36:47.924 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T06:36:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T06:36:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T06:36:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.num_levels: 7 2023-11-15T06:36:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T06:36:47.925 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T06:36:47.926 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T06:36:47.926 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T06:36:47.926 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T06:36:47.926 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T06:36:47.926 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T06:36:47.927 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:36:47.927 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T06:36:47.927 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T06:36:47.927 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T06:36:47.927 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T06:36:47.928 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T06:36:47.928 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:36:47.928 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T06:36:47.928 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T06:36:47.928 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T06:36:47.929 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T06:36:47.929 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T06:36:47.929 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T06:36:47.929 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T06:36:47.929 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T06:36:47.930 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T06:36:47.930 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T06:36:47.930 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T06:36:47.930 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T06:36:47.931 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T06:36:47.931 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T06:36:47.931 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T06:36:47.931 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T06:36:47.931 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T06:36:47.932 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T06:36:47.932 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T06:36:47.932 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T06:36:47.932 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T06:36:47.932 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T06:36:47.933 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T06:36:47.933 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T06:36:47.933 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T06:36:47.933 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T06:36:47.933 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T06:36:47.934 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T06:36:47.934 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T06:36:47.934 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T06:36:47.934 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T06:36:47.934 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T06:36:47.935 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T06:36:47.935 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T06:36:47.935 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T06:36:47.935 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T06:36:47.935 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T06:36:47.935 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T06:36:47.936 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T06:36:47.936 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T06:36:47.936 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T06:36:47.936 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T06:36:47.936 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T06:36:47.937 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T06:36:47.937 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T06:36:47.937 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.ttl: 2592000 2023-11-15T06:36:47.937 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T06:36:47.937 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T06:36:47.938 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.938 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.857+0000 7f626667c700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2023-11-15T06:36:47.938 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.938 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.858+0000 7f626667c700 4 rocksdb: DB pointer 0x55ba17ee1800 2023-11-15T06:36:47.939 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.858+0000 7f624f515700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T06:36:47.939 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.858+0000 7f624f515700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T06:36:47.939 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2023-11-15T06:36:47.939 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:36:47.939 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T06:36:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T06:36:47.940 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:36:47.941 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T06:36:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T06:36:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:36:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:36:47.942 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:36:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:36:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.943 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:36:47.944 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:47.944 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:47.944 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:36:47.944 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:47.944 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:47.945 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:36:47.945 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:36:47.945 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:36:47.945 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:47.945 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:47.945 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:36:47.946 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.946 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:36:47.946 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.946 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:36:47.946 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:47.947 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:47.947 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:36:47.947 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:36:47.947 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.947 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2023-11-15T06:36:47.948 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:47.948 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:47.948 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:36:47.948 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:47.948 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:47.948 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:36:47.949 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:36:47.949 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:36:47.949 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:47.949 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:47.949 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:36:47.950 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.950 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:36:47.950 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: 2023-11-15T06:36:47.950 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.859+0000 7f626667c700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2023-11-15T06:36:47.951 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.859+0000 7f626667c700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2023-11-15T06:36:47.951 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph-mon: debug 2023-11-15T06:36:47.859+0000 7f626667c700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi123 for mon.smithi123 2023-11-15T06:36:48.216 INFO:teuthology.orchestra.run.smithi123.stderr:create mon.smithi123 on 2023-11-15T06:36:48.216 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:48.221 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl disable ceph.target 2023-11-15T06:36:48.228 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2023-11-15T06:36:48.311 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl enable ceph.target 2023-11-15T06:36:48.318 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2023-11-15T06:36:48.399 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl start ceph.target 2023-11-15T06:36:48.405 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl enable ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target 2023-11-15T06:36:48.411 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target. 2023-11-15T06:36:48.411 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target → /etc/systemd/system/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target. 2023-11-15T06:36:48.490 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl start ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target 2023-11-15T06:36:48.496 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl daemon-reload 2023-11-15T06:36:48.580 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123 2023-11-15T06:36:48.603 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl reset-failed ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123 2023-11-15T06:36:48.609 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Failed to reset failed state of unit ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123.service: Unit ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123.service not loaded. 2023-11-15T06:36:48.609 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl enable ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123 2023-11-15T06:36:48.616 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Created symlink /etc/systemd/system/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target.wants/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123.service → /etc/systemd/system/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T06:36:48.702 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl start ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123 2023-11-15T06:36:49.018 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:36:49.028 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: disabled 2023-11-15T06:36:49.028 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:36:49.037 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: inactive 2023-11-15T06:36:49.037 INFO:teuthology.orchestra.run.smithi123.stderr:firewalld.service is not enabled 2023-11-15T06:36:49.038 INFO:teuthology.orchestra.run.smithi123.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T06:36:49.038 INFO:teuthology.orchestra.run.smithi123.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T06:36:49.038 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for mon to start... 2023-11-15T06:36:49.038 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for mon... 2023-11-15T06:36:49.039 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/mon.smithi123:/var/lib/ceph/mon/ceph-smithi123:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2023-11-15T06:36:49.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: cluster 2023-11-15T06:36:49.019558+0000 mon.smithi123 (mon.0) 1 : cluster [INF] mon.smithi123 is new leader, mons smithi123 in quorum (ranks 0) 2023-11-15T06:36:49.467 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2edb0700 1 Processor -- start 2023-11-15T06:36:49.467 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2edb0700 1 -- start start 2023-11-15T06:36:49.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2edb0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c280fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:49.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2edb0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c280fc5b0 con 0x7f1c280fbca0 2023-11-15T06:36:49.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2cb4c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c280fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:49.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2cb4c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c280fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45878/0 (socket says 172.21.15.123:45878) 2023-11-15T06:36:49.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2cb4c700 1 -- 172.21.15.123:0/4120322859 learned_addr learned my addr 172.21.15.123:0/4120322859 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:49.469 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2cb4c700 1 -- 172.21.15.123:0/4120322859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c280fc6f0 con 0x7f1c280fbca0 2023-11-15T06:36:49.469 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.459+0000 7f1c2cb4c700 1 --2- 172.21.15.123:0/4120322859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c280fc0c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1c1c00b5f0 tx=0x7f1c1c006ee0).ready entity=mon.0 client_cookie=703aa4d7f34fe9cd server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:49.469 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c277fe700 1 -- 172.21.15.123:0/4120322859 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c1c00d600 con 0x7f1c280fbca0 2023-11-15T06:36:49.469 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c277fe700 1 -- 172.21.15.123:0/4120322859 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c00d760 con 0x7f1c280fbca0 2023-11-15T06:36:49.469 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c277fe700 1 -- 172.21.15.123:0/4120322859 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c1c00da30 con 0x7f1c280fbca0 2023-11-15T06:36:49.469 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c2edb0700 1 -- 172.21.15.123:0/4120322859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 msgr2=0x7f1c280fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:49.470 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c2edb0700 1 --2- 172.21.15.123:0/4120322859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c280fc0c0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1c1c00b5f0 tx=0x7f1c1c006ee0).stop 2023-11-15T06:36:49.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c2edb0700 1 -- 172.21.15.123:0/4120322859 shutdown_connections 2023-11-15T06:36:49.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c2edb0700 1 --2- 172.21.15.123:0/4120322859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c280fc0c0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:49.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c2edb0700 1 -- 172.21.15.123:0/4120322859 >> 172.21.15.123:0/4120322859 conn(0x7f1c280f74c0 msgr2=0x7f1c280f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:49.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c2edb0700 1 -- 172.21.15.123:0/4120322859 shutdown_connections 2023-11-15T06:36:49.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.460+0000 7f1c2edb0700 1 -- 172.21.15.123:0/4120322859 wait complete. 2023-11-15T06:36:49.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.461+0000 7f1c2edb0700 1 Processor -- start 2023-11-15T06:36:49.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.461+0000 7f1c2edb0700 1 -- start start 2023-11-15T06:36:49.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.461+0000 7f1c2edb0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c2810a270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:49.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.461+0000 7f1c2edb0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c2810a760 con 0x7f1c280fbca0 2023-11-15T06:36:49.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.461+0000 7f1c2cb4c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c2810a270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:49.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.461+0000 7f1c2cb4c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c2810a270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45886/0 (socket says 172.21.15.123:45886) 2023-11-15T06:36:49.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.461+0000 7f1c2cb4c700 1 -- 172.21.15.123:0/762338658 learned_addr learned my addr 172.21.15.123:0/762338658 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:49.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c2cb4c700 1 -- 172.21.15.123:0/762338658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c1c00b200 con 0x7f1c280fbca0 2023-11-15T06:36:49.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c2cb4c700 1 --2- 172.21.15.123:0/762338658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c2810a270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1c1c006c70 tx=0x7f1c1c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:49.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c257fa700 1 -- 172.21.15.123:0/762338658 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c1c011b40 con 0x7f1c280fbca0 2023-11-15T06:36:49.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c257fa700 1 -- 172.21.15.123:0/762338658 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1c1c011ca0 con 0x7f1c280fbca0 2023-11-15T06:36:49.474 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c2810a960 con 0x7f1c280fbca0 2023-11-15T06:36:49.474 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c257fa700 1 -- 172.21.15.123:0/762338658 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c1c01d430 con 0x7f1c280fbca0 2023-11-15T06:36:49.474 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c2810ad70 con 0x7f1c280fbca0 2023-11-15T06:36:49.474 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c257fa700 1 -- 172.21.15.123:0/762338658 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1c1c00f070 con 0x7f1c280fbca0 2023-11-15T06:36:49.474 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.462+0000 7f1c257fa700 1 -- 172.21.15.123:0/762338658 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1c1c027460 con 0x7f1c280fbca0 2023-11-15T06:36:49.475 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.463+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c28045ad0 con 0x7f1c280fbca0 2023-11-15T06:36:49.475 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.466+0000 7f1c257fa700 1 -- 172.21.15.123:0/762338658 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1c1c016070 con 0x7f1c280fbca0 2023-11-15T06:36:49.536 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.531+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7f1c28103d10 con 0x7f1c280fbca0 2023-11-15T06:36:49.537 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.531+0000 7f1c257fa700 1 -- 172.21.15.123:0/762338658 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7f1c1c01d620 con 0x7f1c280fbca0 2023-11-15T06:36:49.537 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: cluster: 2023-11-15T06:36:49.537 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: id: 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:49.537 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: health: HEALTH_OK 2023-11-15T06:36:49.538 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.538 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: services: 2023-11-15T06:36:49.538 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi123 (age 0.49707s) 2023-11-15T06:36:49.538 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mgr: no daemons active 2023-11-15T06:36:49.538 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2023-11-15T06:36:49.538 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.539 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: data: 2023-11-15T06:36:49.539 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2023-11-15T06:36:49.539 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2023-11-15T06:36:49.539 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2023-11-15T06:36:49.540 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: pgs: 2023-11-15T06:36:49.540 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.540 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.532+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 msgr2=0x7f1c2810a270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:49.540 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.532+0000 7f1c2edb0700 1 --2- 172.21.15.123:0/762338658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c2810a270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1c1c006c70 tx=0x7f1c1c016000).stop 2023-11-15T06:36:49.540 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.532+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 shutdown_connections 2023-11-15T06:36:49.540 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.532+0000 7f1c2edb0700 1 --2- 172.21.15.123:0/762338658 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c280fbca0 0x7f1c2810a270 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:49.541 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.532+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 >> 172.21.15.123:0/762338658 conn(0x7f1c280f74c0 msgr2=0x7f1c280fea20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:49.541 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.532+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 shutdown_connections 2023-11-15T06:36:49.541 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:49.532+0000 7f1c2edb0700 1 -- 172.21.15.123:0/762338658 wait complete. 2023-11-15T06:36:49.883 INFO:teuthology.orchestra.run.smithi123.stderr:mon is available 2023-11-15T06:36:49.883 INFO:teuthology.orchestra.run.smithi123.stderr:Assimilating anything we can from ceph.conf... 2023-11-15T06:36:49.883 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z -v /var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/mon.smithi123:/var/lib/ceph/mon/ceph-smithi123:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi123/config 2023-11-15T06:36:50.077 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: cluster 2023-11-15T06:36:49.036146+0000 mon.smithi123 (mon.0) 2 : cluster [INF] mon.smithi123 is new leader, mons smithi123 in quorum (ranks 0) 2023-11-15T06:36:50.078 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: cluster 2023-11-15T06:36:49.036372+0000 mon.smithi123 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0]} 2023-11-15T06:36:50.078 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: cluster 2023-11-15T06:36:49.039641+0000 mon.smithi123 (mon.0) 4 : cluster [DBG] fsmap 2023-11-15T06:36:50.078 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: cluster 2023-11-15T06:36:49.043502+0000 mon.smithi123 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T06:36:50.078 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: cluster 2023-11-15T06:36:49.044029+0000 mon.smithi123 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T06:36:50.079 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: audit 2023-11-15T06:36:49.533178+0000 mon.smithi123 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.123:0/762338658' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2023-11-15T06:36:50.312 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d0b614700 1 Processor -- start 2023-11-15T06:36:50.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d0b614700 1 -- start start 2023-11-15T06:36:50.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d0b614700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d040fe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:50.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d0b614700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d040fedf0 con 0x7f1d04070410 2023-11-15T06:36:50.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d093b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d040fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:50.314 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d093b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d040fe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45900/0 (socket says 172.21.15.123:45900) 2023-11-15T06:36:50.314 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d093b0700 1 -- 172.21.15.123:0/1183640781 learned_addr learned my addr 172.21.15.123:0/1183640781 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:50.316 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d093b0700 1 -- 172.21.15.123:0/1183640781 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1d040fef30 con 0x7f1d04070410 2023-11-15T06:36:50.316 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1d093b0700 1 --2- 172.21.15.123:0/1183640781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d040fe900 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1cf400b5f0 tx=0x7f1cf4006ee0).ready entity=mon.0 client_cookie=7dca65c7ee988d16 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:50.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1cf3fff700 1 -- 172.21.15.123:0/1183640781 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf400d600 con 0x7f1d04070410 2023-11-15T06:36:50.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1cf3fff700 1 -- 172.21.15.123:0/1183640781 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf400d760 con 0x7f1d04070410 2023-11-15T06:36:50.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.309+0000 7f1cf3fff700 1 -- 172.21.15.123:0/1183640781 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf400da30 con 0x7f1d04070410 2023-11-15T06:36:50.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.310+0000 7f1d0b614700 1 -- 172.21.15.123:0/1183640781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 msgr2=0x7f1d040fe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:50.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.310+0000 7f1d0b614700 1 --2- 172.21.15.123:0/1183640781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d040fe900 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f1cf400b5f0 tx=0x7f1cf4006ee0).stop 2023-11-15T06:36:50.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.310+0000 7f1d0b614700 1 -- 172.21.15.123:0/1183640781 shutdown_connections 2023-11-15T06:36:50.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.310+0000 7f1d0b614700 1 --2- 172.21.15.123:0/1183640781 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d040fe900 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:50.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.310+0000 7f1d0b614700 1 -- 172.21.15.123:0/1183640781 >> 172.21.15.123:0/1183640781 conn(0x7f1d040f63c0 msgr2=0x7f1d040f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:50.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.310+0000 7f1d0b614700 1 -- 172.21.15.123:0/1183640781 shutdown_connections 2023-11-15T06:36:50.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.310+0000 7f1d0b614700 1 -- 172.21.15.123:0/1183640781 wait complete. 2023-11-15T06:36:50.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d0b614700 1 Processor -- start 2023-11-15T06:36:50.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d0b614700 1 -- start start 2023-11-15T06:36:50.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d0b614700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d0410a280 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:50.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d0b614700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1d0410a770 con 0x7f1d04070410 2023-11-15T06:36:50.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d093b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d0410a280 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:50.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d093b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d0410a280 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45902/0 (socket says 172.21.15.123:45902) 2023-11-15T06:36:50.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d093b0700 1 -- 172.21.15.123:0/862104120 learned_addr learned my addr 172.21.15.123:0/862104120 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:50.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d093b0700 1 -- 172.21.15.123:0/862104120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cf400b200 con 0x7f1d04070410 2023-11-15T06:36:50.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1d093b0700 1 --2- 172.21.15.123:0/862104120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d0410a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1cf4006c70 tx=0x7f1cf4016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:50.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf4011b40 con 0x7f1d04070410 2023-11-15T06:36:50.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.311+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f1cf4011ca0 con 0x7f1d04070410 2023-11-15T06:36:50.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.312+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1d0410a970 con 0x7f1d04070410 2023-11-15T06:36:50.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.312+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cf401d430 con 0x7f1d04070410 2023-11-15T06:36:50.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.312+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1d0410ad80 con 0x7f1d04070410 2023-11-15T06:36:50.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.312+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1cf400f070 con 0x7f1d04070410 2023-11-15T06:36:50.322 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.312+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1cf4027460 con 0x7f1d04070410 2023-11-15T06:36:50.322 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.313+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1d04045ad0 con 0x7f1d04070410 2023-11-15T06:36:50.322 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.316+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1cf4016070 con 0x7f1d04070410 2023-11-15T06:36:50.382 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.379+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f1d040ff180 con 0x7f1d04070410 2023-11-15T06:36:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.382+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1cf401d5e0 con 0x7f1d04070410 2023-11-15T06:36:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.382+0000 7f1cf1ffb700 1 -- 172.21.15.123:0/862104120 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f1cf4036020 con 0x7f1d04070410 2023-11-15T06:36:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.388 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: [global] 2023-11-15T06:36:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: fsid = 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.123:3300,v1:172.21.15.123:6789] 2023-11-15T06:36:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2023-11-15T06:36:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2023-11-15T06:36:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2023-11-15T06:36:50.389 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2023-11-15T06:36:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: [mgr] 2023-11-15T06:36:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2023-11-15T06:36:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.390 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: [osd] 2023-11-15T06:36:50.391 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2023-11-15T06:36:50.391 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2023-11-15T06:36:50.391 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2023-11-15T06:36:50.391 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2023-11-15T06:36:50.391 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2023-11-15T06:36:50.392 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2023-11-15T06:36:50.392 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.383+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 msgr2=0x7f1d0410a280 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:50.393 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.383+0000 7f1d0b614700 1 --2- 172.21.15.123:0/862104120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d0410a280 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f1cf4006c70 tx=0x7f1cf4016000).stop 2023-11-15T06:36:50.394 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.383+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 shutdown_connections 2023-11-15T06:36:50.394 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.383+0000 7f1d0b614700 1 --2- 172.21.15.123:0/862104120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1d04070410 0x7f1d0410a280 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:50.395 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.383+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 >> 172.21.15.123:0/862104120 conn(0x7f1d040f63c0 msgr2=0x7f1d040f8100 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:50.395 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.383+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 shutdown_connections 2023-11-15T06:36:50.396 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:50.383+0000 7f1d0b614700 1 -- 172.21.15.123:0/862104120 wait complete. 2023-11-15T06:36:50.741 INFO:teuthology.orchestra.run.smithi123.stderr:Generating new minimal ceph.conf... 2023-11-15T06:36:50.742 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z -v /var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/mon.smithi123:/var/lib/ceph/mon/ceph-smithi123:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi123/config 2023-11-15T06:36:51.136 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909759e700 1 Processor -- start 2023-11-15T06:36:51.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909759e700 1 -- start start 2023-11-15T06:36:51.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909759e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f90980fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:51.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909759e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9098070790 con 0x7f9098070370 2023-11-15T06:36:51.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f90980fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f90980fe7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45906/0 (socket says 172.21.15.123:45906) 2023-11-15T06:36:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909659c700 1 -- 172.21.15.123:0/4257534145 learned_addr learned my addr 172.21.15.123:0/4257534145 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.132+0000 7f909659c700 1 -- 172.21.15.123:0/4257534145 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f90980fece0 con 0x7f9098070370 2023-11-15T06:36:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909659c700 1 --2- 172.21.15.123:0/4257534145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f90980fe7f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f908800b5f0 tx=0x7f9088006ee0).ready entity=mon.0 client_cookie=7a062f52f002ec67 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:51.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909559a700 1 -- 172.21.15.123:0/4257534145 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908800d600 con 0x7f9098070370 2023-11-15T06:36:51.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909559a700 1 -- 172.21.15.123:0/4257534145 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f908800dbd0 con 0x7f9098070370 2023-11-15T06:36:51.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909559a700 1 -- 172.21.15.123:0/4257534145 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9088011c00 con 0x7f9098070370 2023-11-15T06:36:51.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909759e700 1 -- 172.21.15.123:0/4257534145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 msgr2=0x7f90980fe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:51.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909759e700 1 --2- 172.21.15.123:0/4257534145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f90980fe7f0 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f908800b5f0 tx=0x7f9088006ee0).stop 2023-11-15T06:36:51.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909759e700 1 -- 172.21.15.123:0/4257534145 shutdown_connections 2023-11-15T06:36:51.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909759e700 1 --2- 172.21.15.123:0/4257534145 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f90980fe7f0 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:51.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909759e700 1 -- 172.21.15.123:0/4257534145 >> 172.21.15.123:0/4257534145 conn(0x7f90980f6210 msgr2=0x7f90980f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:51.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909759e700 1 -- 172.21.15.123:0/4257534145 shutdown_connections 2023-11-15T06:36:51.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.133+0000 7f909759e700 1 -- 172.21.15.123:0/4257534145 wait complete. 2023-11-15T06:36:51.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909759e700 1 Processor -- start 2023-11-15T06:36:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909759e700 1 -- start start 2023-11-15T06:36:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909759e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f909810a850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:51.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909759e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f909810ad40 con 0x7f9098070370 2023-11-15T06:36:51.143 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f909810a850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:51.143 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f909810a850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45914/0 (socket says 172.21.15.123:45914) 2023-11-15T06:36:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909659c700 1 -- 172.21.15.123:0/262560697 learned_addr learned my addr 172.21.15.123:0/262560697 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.134+0000 7f909659c700 1 -- 172.21.15.123:0/262560697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f908800b200 con 0x7f9098070370 2023-11-15T06:36:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f909659c700 1 --2- 172.21.15.123:0/262560697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f909810a850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f908800e040 tx=0x7f9088006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f9086ffd700 1 -- 172.21.15.123:0/262560697 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908800d9f0 con 0x7f9098070370 2023-11-15T06:36:51.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f9086ffd700 1 -- 172.21.15.123:0/262560697 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f908801a9f0 con 0x7f9098070370 2023-11-15T06:36:51.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f909810af40 con 0x7f9098070370 2023-11-15T06:36:51.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f9086ffd700 1 -- 172.21.15.123:0/262560697 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f908801b890 con 0x7f9098070370 2023-11-15T06:36:51.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f909810b350 con 0x7f9098070370 2023-11-15T06:36:51.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f9086ffd700 1 -- 172.21.15.123:0/262560697 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f908800f070 con 0x7f9098070370 2023-11-15T06:36:51.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.135+0000 7f9086ffd700 1 -- 172.21.15.123:0/262560697 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9088025cc0 con 0x7f9098070370 2023-11-15T06:36:51.146 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.136+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9098045ad0 con 0x7f9098070370 2023-11-15T06:36:51.146 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.138+0000 7f9086ffd700 1 -- 172.21.15.123:0/262560697 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9088016070 con 0x7f9098070370 2023-11-15T06:36:51.204 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.201+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f90980ff960 con 0x7f9098070370 2023-11-15T06:36:51.239 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.201+0000 7f9086ffd700 1 -- 172.21.15.123:0/262560697 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f9088025460 con 0x7f9098070370 2023-11-15T06:36:51.240 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.202+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 msgr2=0x7f909810a850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:51.240 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.202+0000 7f909759e700 1 --2- 172.21.15.123:0/262560697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f909810a850 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f908800e040 tx=0x7f9088006ee0).stop 2023-11-15T06:36:51.241 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.202+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 shutdown_connections 2023-11-15T06:36:51.241 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.202+0000 7f909759e700 1 --2- 172.21.15.123:0/262560697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9098070370 0x7f909810a850 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:51.241 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.202+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 >> 172.21.15.123:0/262560697 conn(0x7f90980f6210 msgr2=0x7f90980f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:51.241 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.202+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 shutdown_connections 2023-11-15T06:36:51.242 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:51.202+0000 7f909759e700 1 -- 172.21.15.123:0/262560697 wait complete. 2023-11-15T06:36:51.373 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: audit 2023-11-15T06:36:50.381212+0000 mon.smithi123 (mon.0) 8 : audit [INF] from='client.? 172.21.15.123:0/862104120' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T06:36:51.373 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: audit 2023-11-15T06:36:50.384293+0000 mon.smithi123 (mon.0) 9 : audit [INF] from='client.? 172.21.15.123:0/862104120' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T06:36:51.558 INFO:teuthology.orchestra.run.smithi123.stderr:Restarting the monitor... 2023-11-15T06:36:51.559 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl restart ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123 2023-11-15T06:36:51.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:51 smithi123 systemd[1]: Stopping Ceph mon.smithi123 for 4a6dbac0-8381-11ee-8dbe-212e2dc638e7... 2023-11-15T06:36:51.973 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: debug 2023-11-15T06:36:51.654+0000 7f81b1a92700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi123 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T06:36:51.974 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[111902]: debug 2023-11-15T06:36:51.654+0000 7f81b1a92700 -1 mon.smithi123@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T06:36:52.365 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 podman[112222]: ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123 2023-11-15T06:36:52.366 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 systemd[1]: ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123.service: Succeeded. 2023-11-15T06:36:52.366 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 systemd[1]: Stopped Ceph mon.smithi123 for 4a6dbac0-8381-11ee-8dbe-212e2dc638e7. 2023-11-15T06:36:52.366 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 systemd[1]: Starting Ceph mon.smithi123 for 4a6dbac0-8381-11ee-8dbe-212e2dc638e7... 2023-11-15T06:36:52.450 INFO:teuthology.orchestra.run.smithi123.stderr:Setting mon public_network to 172.21.0.0/20 2023-11-15T06:36:52.451 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2023-11-15T06:36:52.808 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.802+0000 7fe802b43700 1 Processor -- start 2023-11-15T06:36:52.808 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.802+0000 7fe802b43700 1 -- start start 2023-11-15T06:36:52.808 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.802+0000 7fe802b43700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:52.808 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.802+0000 7fe802b43700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc0fb490 con 0x7fe7fc0fab80 2023-11-15T06:36:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe8008df700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe8008df700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45922/0 (socket says 172.21.15.123:45922) 2023-11-15T06:36:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe8008df700 1 -- 172.21.15.123:0/3393818354 learned_addr learned my addr 172.21.15.123:0/3393818354 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe8008df700 1 -- 172.21.15.123:0/3393818354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7fc0fb5d0 con 0x7fe7fc0fab80 2023-11-15T06:36:52.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe8008df700 1 --2- 172.21.15.123:0/3393818354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc0fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe7f4009960 tx=0x7fe7f40092d0).ready entity=mon.0 client_cookie=d89b1ac7fb8d847c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe7fb7fe700 1 -- 172.21.15.123:0/3393818354 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f400b9c0 con 0x7fe7fc0fab80 2023-11-15T06:36:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe7fb7fe700 1 -- 172.21.15.123:0/3393818354 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe7f4011400 con 0x7fe7fc0fab80 2023-11-15T06:36:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.803+0000 7fe7fb7fe700 1 -- 172.21.15.123:0/3393818354 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f40103a0 con 0x7fe7fc0fab80 2023-11-15T06:36:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.804+0000 7fe802b43700 1 -- 172.21.15.123:0/3393818354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 msgr2=0x7fe7fc0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:52.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.804+0000 7fe802b43700 1 --2- 172.21.15.123:0/3393818354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc0fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fe7f4009960 tx=0x7fe7f40092d0).stop 2023-11-15T06:36:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.804+0000 7fe802b43700 1 -- 172.21.15.123:0/3393818354 shutdown_connections 2023-11-15T06:36:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.804+0000 7fe802b43700 1 --2- 172.21.15.123:0/3393818354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc0fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.804+0000 7fe802b43700 1 -- 172.21.15.123:0/3393818354 >> 172.21.15.123:0/3393818354 conn(0x7fe7fc0f63c0 msgr2=0x7fe7fc0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.804+0000 7fe802b43700 1 -- 172.21.15.123:0/3393818354 shutdown_connections 2023-11-15T06:36:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.804+0000 7fe802b43700 1 -- 172.21.15.123:0/3393818354 wait complete. 2023-11-15T06:36:52.811 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe802b43700 1 Processor -- start 2023-11-15T06:36:52.812 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe802b43700 1 -- start start 2023-11-15T06:36:52.812 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe802b43700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc10d6e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:52.813 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe802b43700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe7fc070470 con 0x7fe7fc0fab80 2023-11-15T06:36:52.813 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe8008df700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc10d6e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:52.813 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe8008df700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc10d6e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45932/0 (socket says 172.21.15.123:45932) 2023-11-15T06:36:52.814 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe8008df700 1 -- 172.21.15.123:0/1242902049 learned_addr learned my addr 172.21.15.123:0/1242902049 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:52.814 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe8008df700 1 -- 172.21.15.123:0/1242902049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe7f4009570 con 0x7fe7fc0fab80 2023-11-15T06:36:52.814 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.805+0000 7fe8008df700 1 --2- 172.21.15.123:0/1242902049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc10d6e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe7f400ba90 tx=0x7fe7f40092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:52.814 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.806+0000 7fe7f97fa700 1 -- 172.21.15.123:0/1242902049 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f4010870 con 0x7fe7fc0fab80 2023-11-15T06:36:52.814 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.806+0000 7fe7f97fa700 1 -- 172.21.15.123:0/1242902049 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe7f4010e40 con 0x7fe7fc0fab80 2023-11-15T06:36:52.815 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.806+0000 7fe7f97fa700 1 -- 172.21.15.123:0/1242902049 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe7f401ae10 con 0x7fe7fc0fab80 2023-11-15T06:36:52.815 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.806+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe7fc070670 con 0x7fe7fc0fab80 2023-11-15T06:36:52.815 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.806+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe7fc10dd90 con 0x7fe7fc0fab80 2023-11-15T06:36:52.815 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.806+0000 7fe7f97fa700 1 -- 172.21.15.123:0/1242902049 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fe7f402a070 con 0x7fe7fc0fab80 2023-11-15T06:36:52.816 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.806+0000 7fe7f97fa700 1 -- 172.21.15.123:0/1242902049 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fe7f4022780 con 0x7fe7fc0fab80 2023-11-15T06:36:52.816 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.807+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe7fc045ad0 con 0x7fe7fc0fab80 2023-11-15T06:36:52.816 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.809+0000 7fe7f97fa700 1 -- 172.21.15.123:0/1242902049 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fe7f4028060 con 0x7fe7fc0fab80 2023-11-15T06:36:52.874 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.872+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7fe7fc027a00 con 0x7fe7fc0fab80 2023-11-15T06:36:52.877 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.874+0000 7fe7f97fa700 1 -- 172.21.15.123:0/1242902049 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe7f401e030 con 0x7fe7fc0fab80 2023-11-15T06:36:52.878 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.875+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 msgr2=0x7fe7fc10d6e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:52.878 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.875+0000 7fe802b43700 1 --2- 172.21.15.123:0/1242902049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc10d6e0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fe7f400ba90 tx=0x7fe7f40092d0).stop 2023-11-15T06:36:52.878 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.875+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 shutdown_connections 2023-11-15T06:36:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.875+0000 7fe802b43700 1 --2- 172.21.15.123:0/1242902049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe7fc0fab80 0x7fe7fc10d6e0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.875+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 >> 172.21.15.123:0/1242902049 conn(0x7fe7fc0f63c0 msgr2=0x7fe7fc0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.875+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 shutdown_connections 2023-11-15T06:36:52.879 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:52.875+0000 7fe802b43700 1 -- 172.21.15.123:0/1242902049 wait complete. 2023-11-15T06:36:52.899 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.438+0000 7fd860215700 0 set uid:gid to 167:167 (ceph:ceph) 2023-11-15T06:36:52.899 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.438+0000 7fd860215700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2023-11-15T06:36:52.900 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.438+0000 7fd860215700 0 pidfile_write: ignore empty --pid-file 2023-11-15T06:36:52.900 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.442+0000 7fd860215700 0 load: jerasure load: lrc load: isa 2023-11-15T06:36:52.900 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: RocksDB version: 6.8.1 2023-11-15T06:36:52.900 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.901 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2023-11-15T06:36:52.901 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Compile date May 13 2021 2023-11-15T06:36:52.901 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: DB SUMMARY 2023-11-15T06:36:52.901 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.901 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: CURRENT file: CURRENT 2023-11-15T06:36:52.901 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.902 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: IDENTITY file: IDENTITY 2023-11-15T06:36:52.902 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.902 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2023-11-15T06:36:52.902 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.902 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi123/store.db dir, Total Num: 1, files: 000004.sst 2023-11-15T06:36:52.903 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.903 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi123/store.db: 000006.log size: 92930 ; 2023-11-15T06:36:52.903 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.903 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.error_if_exists: 0 2023-11-15T06:36:52.903 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.create_if_missing: 0 2023-11-15T06:36:52.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.paranoid_checks: 1 2023-11-15T06:36:52.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.env: 0x55ad832211c0 2023-11-15T06:36:52.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.fs: Posix File System 2023-11-15T06:36:52.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.info_log: 0x55ad84409960 2023-11-15T06:36:52.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_file_opening_threads: 16 2023-11-15T06:36:52.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.statistics: (nil) 2023-11-15T06:36:52.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.use_fsync: 0 2023-11-15T06:36:52.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_log_file_size: 0 2023-11-15T06:36:52.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2023-11-15T06:36:52.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.log_file_time_to_roll: 0 2023-11-15T06:36:52.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.keep_log_file_num: 1000 2023-11-15T06:36:52.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.recycle_log_file_num: 0 2023-11-15T06:36:52.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.allow_fallocate: 1 2023-11-15T06:36:52.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.allow_mmap_reads: 0 2023-11-15T06:36:52.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.allow_mmap_writes: 0 2023-11-15T06:36:52.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.use_direct_reads: 0 2023-11-15T06:36:52.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2023-11-15T06:36:52.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.create_missing_column_families: 0 2023-11-15T06:36:52.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.db_log_dir: 2023-11-15T06:36:52.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 bash[112292]: 371a3c0b165dcafa35bd67265ce170365fab91a3108d83431874e7bc275192b0 2023-11-15T06:36:52.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi123/store.db 2023-11-15T06:36:52.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.table_cache_numshardbits: 6 2023-11-15T06:36:52.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_subcompactions: 1 2023-11-15T06:36:52.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_background_flushes: -1 2023-11-15T06:36:52.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.WAL_ttl_seconds: 0 2023-11-15T06:36:52.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.WAL_size_limit_MB: 0 2023-11-15T06:36:52.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2023-11-15T06:36:52.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2023-11-15T06:36:52.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.is_fd_close_on_exec: 1 2023-11-15T06:36:52.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.advise_random_on_open: 1 2023-11-15T06:36:52.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.db_write_buffer_size: 0 2023-11-15T06:36:52.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.write_buffer_manager: 0x55ad8440ffb0 2023-11-15T06:36:52.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2023-11-15T06:36:52.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2023-11-15T06:36:52.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2023-11-15T06:36:52.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.use_adaptive_mutex: 0 2023-11-15T06:36:52.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.rate_limiter: (nil) 2023-11-15T06:36:52.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2023-11-15T06:36:52.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.wal_recovery_mode: 2 2023-11-15T06:36:52.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.enable_thread_tracking: 0 2023-11-15T06:36:52.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.enable_pipelined_write: 0 2023-11-15T06:36:52.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.unordered_write: 0 2023-11-15T06:36:52.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2023-11-15T06:36:52.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2023-11-15T06:36:52.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2023-11-15T06:36:52.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2023-11-15T06:36:52.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.row_cache: None 2023-11-15T06:36:52.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.wal_filter: None 2023-11-15T06:36:52.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2023-11-15T06:36:52.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.allow_ingest_behind: 0 2023-11-15T06:36:52.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.preserve_deletes: 0 2023-11-15T06:36:52.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.two_write_queues: 0 2023-11-15T06:36:52.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.manual_wal_flush: 0 2023-11-15T06:36:52.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.atomic_flush: 0 2023-11-15T06:36:52.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2023-11-15T06:36:52.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.persist_stats_to_disk: 0 2023-11-15T06:36:52.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.write_dbid_to_manifest: 0 2023-11-15T06:36:52.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.log_readahead_size: 0 2023-11-15T06:36:52.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2023-11-15T06:36:52.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_background_jobs: 2 2023-11-15T06:36:52.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_background_compactions: -1 2023-11-15T06:36:52.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2023-11-15T06:36:52.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2023-11-15T06:36:52.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.delayed_write_rate : 16777216 2023-11-15T06:36:52.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_total_wal_size: 0 2023-11-15T06:36:52.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2023-11-15T06:36:52.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.stats_dump_period_sec: 600 2023-11-15T06:36:52.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.stats_persist_period_sec: 600 2023-11-15T06:36:52.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2023-11-15T06:36:52.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_open_files: -1 2023-11-15T06:36:52.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bytes_per_sync: 0 2023-11-15T06:36:52.918 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.wal_bytes_per_sync: 0 2023-11-15T06:36:52.918 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.strict_bytes_per_sync: 0 2023-11-15T06:36:52.918 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_readahead_size: 0 2023-11-15T06:36:52.918 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Compression algorithms supported: 2023-11-15T06:36:52.918 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2023-11-15T06:36:52.918 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kZSTD supported: 0 2023-11-15T06:36:52.919 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kXpressCompression supported: 0 2023-11-15T06:36:52.919 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kLZ4HCCompression supported: 1 2023-11-15T06:36:52.919 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kLZ4Compression supported: 1 2023-11-15T06:36:52.919 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kBZip2Compression supported: 0 2023-11-15T06:36:52.919 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kZlibCompression supported: 1 2023-11-15T06:36:52.920 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: kSnappyCompression supported: 1 2023-11-15T06:36:52.920 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Fast CRC32 supported: Supported on x86 2023-11-15T06:36:52.920 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000005 2023-11-15T06:36:52.920 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.920 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2023-11-15T06:36:52.921 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.921 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2023-11-15T06:36:52.921 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.merge_operator: 2023-11-15T06:36:52.921 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_filter: None 2023-11-15T06:36:52.921 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_filter_factory: None 2023-11-15T06:36:52.921 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.memtable_factory: SkipListFactory 2023-11-15T06:36:52.922 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.table_factory: BlockBasedTable 2023-11-15T06:36:52.922 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x55ad84320128) 2023-11-15T06:36:52.922 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cache_index_and_filter_blocks: 1 2023-11-15T06:36:52.922 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cache_index_and_filter_blocks_with_high_priority: 0 2023-11-15T06:36:52.922 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: pin_l0_filter_and_index_blocks_in_cache: 0 2023-11-15T06:36:52.923 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: pin_top_level_index_and_filter: 1 2023-11-15T06:36:52.923 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: index_type: 0 2023-11-15T06:36:52.923 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: data_block_index_type: 0 2023-11-15T06:36:52.923 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: index_shortening: 1 2023-11-15T06:36:52.923 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: data_block_hash_table_util_ratio: 0.750000 2023-11-15T06:36:52.924 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: hash_index_allow_collision: 1 2023-11-15T06:36:52.924 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: checksum: 1 2023-11-15T06:36:52.924 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: no_block_cache: 0 2023-11-15T06:36:52.924 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_cache: 0x55ad84356f10 2023-11-15T06:36:52.924 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_cache_name: BinnedLRUCache 2023-11-15T06:36:52.924 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_cache_options: 2023-11-15T06:36:52.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: capacity : 536870912 2023-11-15T06:36:52.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: num_shard_bits : 4 2023-11-15T06:36:52.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: strict_capacity_limit : 0 2023-11-15T06:36:52.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: high_pri_pool_ratio: 0.000 2023-11-15T06:36:52.925 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_cache_compressed: (nil) 2023-11-15T06:36:52.926 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: persistent_cache: (nil) 2023-11-15T06:36:52.926 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_size: 4096 2023-11-15T06:36:52.926 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_size_deviation: 10 2023-11-15T06:36:52.926 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_restart_interval: 16 2023-11-15T06:36:52.926 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: index_block_restart_interval: 1 2023-11-15T06:36:52.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: metadata_block_size: 4096 2023-11-15T06:36:52.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: partition_filters: 0 2023-11-15T06:36:52.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: use_delta_encoding: 1 2023-11-15T06:36:52.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: filter_policy: rocksdb.BuiltinBloomFilter 2023-11-15T06:36:52.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: whole_key_filtering: 1 2023-11-15T06:36:52.927 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: verify_compression: 0 2023-11-15T06:36:52.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: read_amp_bytes_per_bit: 0 2023-11-15T06:36:52.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: format_version: 2 2023-11-15T06:36:52.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: enable_index_compression: 1 2023-11-15T06:36:52.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: block_align: 0 2023-11-15T06:36:52.928 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.929 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.write_buffer_size: 33554432 2023-11-15T06:36:52.929 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_write_buffer_number: 2 2023-11-15T06:36:52.929 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compression: NoCompression 2023-11-15T06:36:52.929 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bottommost_compression: Disabled 2023-11-15T06:36:52.929 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.prefix_extractor: nullptr 2023-11-15T06:36:52.929 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2023-11-15T06:36:52.930 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.num_levels: 7 2023-11-15T06:36:52.930 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2023-11-15T06:36:52.930 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2023-11-15T06:36:52.932 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2023-11-15T06:36:52.932 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2023-11-15T06:36:52.932 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2023-11-15T06:36:52.932 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2023-11-15T06:36:52.933 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2023-11-15T06:36:52.933 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:36:52.933 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2023-11-15T06:36:52.933 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compression_opts.window_bits: -14 2023-11-15T06:36:52.934 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compression_opts.level: 32767 2023-11-15T06:36:52.934 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compression_opts.strategy: 0 2023-11-15T06:36:52.934 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2023-11-15T06:36:52.934 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2023-11-15T06:36:52.934 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compression_opts.enabled: false 2023-11-15T06:36:52.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2023-11-15T06:36:52.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2023-11-15T06:36:52.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2023-11-15T06:36:52.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.target_file_size_base: 67108864 2023-11-15T06:36:52.935 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.target_file_size_multiplier: 1 2023-11-15T06:36:52.936 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2023-11-15T06:36:52.936 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2023-11-15T06:36:52.936 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2023-11-15T06:36:52.936 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2023-11-15T06:36:52.936 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2023-11-15T06:36:52.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2023-11-15T06:36:52.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2023-11-15T06:36:52.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2023-11-15T06:36:52.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2023-11-15T06:36:52.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2023-11-15T06:36:52.937 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2023-11-15T06:36:52.938 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2023-11-15T06:36:52.938 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.arena_block_size: 4194304 2023-11-15T06:36:52.938 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2023-11-15T06:36:52.938 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2023-11-15T06:36:52.938 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2023-11-15T06:36:52.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.disable_auto_compactions: 0 2023-11-15T06:36:52.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2023-11-15T06:36:52.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2023-11-15T06:36:52.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2023-11-15T06:36:52.939 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2023-11-15T06:36:52.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2023-11-15T06:36:52.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2023-11-15T06:36:52.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2023-11-15T06:36:52.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2023-11-15T06:36:52.940 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2023-11-15T06:36:52.941 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2023-11-15T06:36:52.941 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.table_properties_collectors: 2023-11-15T06:36:52.941 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.inplace_update_support: 0 2023-11-15T06:36:52.941 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.inplace_update_num_locks: 10000 2023-11-15T06:36:52.941 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2023-11-15T06:36:52.942 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2023-11-15T06:36:52.942 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.memtable_huge_page_size: 0 2023-11-15T06:36:52.942 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.bloom_locality: 0 2023-11-15T06:36:52.942 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.max_successive_merges: 0 2023-11-15T06:36:52.942 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.optimize_filters_for_hits: 0 2023-11-15T06:36:52.943 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.paranoid_file_checks: 0 2023-11-15T06:36:52.943 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.force_consistency_checks: 0 2023-11-15T06:36:52.943 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.report_bg_io_stats: 0 2023-11-15T06:36:52.943 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.ttl: 2592000 2023-11-15T06:36:52.943 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.443+0000 7fd860215700 4 rocksdb: Options.periodic_compaction_seconds: 0 2023-11-15T06:36:52.943 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.444+0000 7fd860215700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi123/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2023-11-15T06:36:52.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.444+0000 7fd860215700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2023-11-15T06:36:52.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.444+0000 7fd860215700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700030212445153, "job": 1, "event": "recovery_started", "log_files": [6]} 2023-11-15T06:36:52.944 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.444+0000 7fd860215700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2023-11-15T06:36:52.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.444+0000 7fd860215700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2023-11-15T06:36:52.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.445+0000 7fd860215700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700030212446070, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88710, "table_properties": {"data_size": 86825, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79966, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1700030212, "oldest_key_time": 3, "file_creation_time": 0}} 2023-11-15T06:36:52.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.445+0000 7fd860215700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2023-11-15T06:36:52.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.945 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.445+0000 7fd860215700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1700030212446488, "job": 1, "event": "recovery_finished"} 2023-11-15T06:36:52.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.446+0000 7fd860215700 4 rocksdb: DB pointer 0x55ad8441b800 2023-11-15T06:36:52.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.446+0000 7fd84702d700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2023-11-15T06:36:52.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.446+0000 7fd84702d700 4 rocksdb: [db_impl/db_impl.cc:851] 2023-11-15T06:36:52.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ** DB Stats ** 2023-11-15T06:36:52.946 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:36:52.947 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2023-11-15T06:36:52.947 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2023-11-15T06:36:52.947 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:36:52.947 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2023-11-15T06:36:52.947 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2023-11-15T06:36:52.947 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2023-11-15T06:36:52.948 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.948 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ** Compaction Stats [default] ** 2023-11-15T06:36:52.948 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:52.948 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:52.948 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: L0 2/0 88.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 2023-11-15T06:36:52.949 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Sum 2/0 88.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 2023-11-15T06:36:52.949 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 2023-11-15T06:36:52.949 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.949 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ** Compaction Stats [default] ** 2023-11-15T06:36:52.949 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:52.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:52.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.9 0.00 0.00 1 0.001 0 0 2023-11-15T06:36:52.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:36:52.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:52.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:52.950 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:36:52.951 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:36:52.951 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:36:52.951 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Cumulative compaction: 0.00 GB write, 29.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:52.951 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Interval compaction: 0.00 GB write, 29.11 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:52.951 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:36:52.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:36:52.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ** Compaction Stats [default] ** 2023-11-15T06:36:52.952 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:52.953 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:52.953 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: L0 2/0 88.32 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 2023-11-15T06:36:52.953 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Sum 2/0 88.32 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 133.9 0.00 0.00 1 0.001 0 0 2023-11-15T06:36:52.953 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2023-11-15T06:36:52.954 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.954 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ** Compaction Stats [default] ** 2023-11-15T06:36:52.954 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2023-11-15T06:36:52.954 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2023-11-15T06:36:52.954 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 133.9 0.00 0.00 1 0.001 0 0 2023-11-15T06:36:52.955 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Uptime(secs): 0.0 total, 0.0 interval 2023-11-15T06:36:52.955 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Flush(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:52.955 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(GB): cumulative 0.000, interval 0.000 2023-11-15T06:36:52.955 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(Total Files): cumulative 0, interval 0 2023-11-15T06:36:52.955 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(L0 Files): cumulative 0, interval 0 2023-11-15T06:36:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: AddFile(Keys): cumulative 0, interval 0 2023-11-15T06:36:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Cumulative compaction: 0.00 GB write, 28.82 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2023-11-15T06:36:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2023-11-15T06:36:52.956 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ** File Read Latency Histogram By Level [default] ** 2023-11-15T06:36:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.447+0000 7fd860215700 0 starting mon.smithi123 rank 0 at public addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] at bind addrs [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi123 fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:52.957 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 systemd[1]: Started Ceph mon.smithi123 for 4a6dbac0-8381-11ee-8dbe-212e2dc638e7. 2023-11-15T06:36:52.958 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 1 mon.smithi123@-1(???) e1 preinit fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:36:52.958 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 0 mon.smithi123@-1(???).mds e1 new map 2023-11-15T06:36:52.958 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 0 mon.smithi123@-1(???).mds e1 print_map 2023-11-15T06:36:52.958 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: e1 2023-11-15T06:36:52.958 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: enable_multiple, ever_enabled_multiple: 1,1 2023-11-15T06:36:52.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2023-11-15T06:36:52.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: legacy client fscid: -1 2023-11-15T06:36:52.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: No filesystems configured 2023-11-15T06:36:52.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: 2023-11-15T06:36:52.959 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 0 mon.smithi123@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2023-11-15T06:36:52.960 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 0 mon.smithi123@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T06:36:52.960 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 0 mon.smithi123@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T06:36:52.960 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 0 mon.smithi123@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2023-11-15T06:36:52.960 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:50.381212+0000 mon.smithi123 (mon.0) 8 : audit [INF] from='client.? 172.21.15.123:0/862104120' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2023-11-15T06:36:52.960 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:50.384293+0000 mon.smithi123 (mon.0) 9 : audit [INF] from='client.? 172.21.15.123:0/862104120' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2023-11-15T06:36:52.961 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.448+0000 7fd860215700 1 mon.smithi123@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2023-11-15T06:36:52.961 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expand map: {default=false} 2023-11-15T06:36:52.961 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta from 'false' to 'false' 2023-11-15T06:36:52.961 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expanded map: {default=false} 2023-11-15T06:36:52.961 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expand map: {default=info} 2023-11-15T06:36:52.962 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta from 'info' to 'info' 2023-11-15T06:36:52.962 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expanded map: {default=info} 2023-11-15T06:36:52.962 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expand map: {default=daemon} 2023-11-15T06:36:52.962 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta from 'daemon' to 'daemon' 2023-11-15T06:36:52.962 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expanded map: {default=daemon} 2023-11-15T06:36:52.962 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expand map: {default=debug} 2023-11-15T06:36:52.963 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta from 'debug' to 'debug' 2023-11-15T06:36:52.963 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:36:52.449+0000 7fd860215700 20 expand_channel_meta expanded map: {default=debug} 2023-11-15T06:36:52.963 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:52.464377+0000 mon.smithi123 (mon.0) 1 : cluster [INF] mon.smithi123 is new leader, mons smithi123 in quorum (ranks 0) 2023-11-15T06:36:52.963 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:52.464437+0000 mon.smithi123 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi123=[v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0]} 2023-11-15T06:36:52.963 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:52.464481+0000 mon.smithi123 (mon.0) 3 : cluster [DBG] fsmap 2023-11-15T06:36:52.964 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:52.464505+0000 mon.smithi123 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2023-11-15T06:36:52.964 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:52.464920+0000 mon.smithi123 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2023-11-15T06:36:53.275 INFO:teuthology.orchestra.run.smithi123.stderr:Wrote config to /etc/ceph/ceph.conf 2023-11-15T06:36:53.275 INFO:teuthology.orchestra.run.smithi123.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2023-11-15T06:36:53.275 INFO:teuthology.orchestra.run.smithi123.stderr:Creating mgr... 2023-11-15T06:36:53.276 INFO:teuthology.orchestra.run.smithi123.stderr:Verifying port 9283 ... 2023-11-15T06:36:53.277 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl daemon-reload 2023-11-15T06:36:53.372 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mgr.smithi123.razjtn 2023-11-15T06:36:53.395 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl reset-failed ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mgr.smithi123.razjtn 2023-11-15T06:36:53.401 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Failed to reset failed state of unit ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mgr.smithi123.razjtn.service: Unit ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mgr.smithi123.razjtn.service not loaded. 2023-11-15T06:36:53.401 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl enable ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mgr.smithi123.razjtn 2023-11-15T06:36:53.407 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: Created symlink /etc/systemd/system/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7.target.wants/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mgr.smithi123.razjtn.service → /etc/systemd/system/ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@.service. 2023-11-15T06:36:53.489 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl start ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mgr.smithi123.razjtn 2023-11-15T06:36:53.817 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:36:53.824 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: disabled 2023-11-15T06:36:53.825 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:36:53.832 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: inactive 2023-11-15T06:36:53.833 INFO:teuthology.orchestra.run.smithi123.stderr:firewalld.service is not enabled 2023-11-15T06:36:53.833 INFO:teuthology.orchestra.run.smithi123.stderr:Not possible to enable service . firewalld.service is not available 2023-11-15T06:36:53.833 INFO:teuthology.orchestra.run.smithi123.stderr:Not possible to open ports <[]>. firewalld.service is not available 2023-11-15T06:36:53.833 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:36:53.840 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: disabled 2023-11-15T06:36:53.841 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:36:53.848 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: inactive 2023-11-15T06:36:53.849 INFO:teuthology.orchestra.run.smithi123.stderr:firewalld.service is not enabled 2023-11-15T06:36:53.849 INFO:teuthology.orchestra.run.smithi123.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2023-11-15T06:36:53.849 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for mgr to start... 2023-11-15T06:36:53.849 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for mgr... 2023-11-15T06:36:53.850 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T06:36:54.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:52.875618+0000 mon.smithi123 (mon.0) 6 : audit [INF] from='client.? 172.21.15.123:0/1242902049' entity='client.admin' 2023-11-15T06:36:54.249 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.245+0000 7f89ff59e700 1 Processor -- start 2023-11-15T06:36:54.250 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.245+0000 7f89ff59e700 1 -- start start 2023-11-15T06:36:54.250 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.245+0000 7f89ff59e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:54.250 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.245+0000 7f89ff59e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000fec20 con 0x7f8a00070430 2023-11-15T06:36:54.250 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fe59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:54.251 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fe59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45950/0 (socket says 172.21.15.123:45950) 2023-11-15T06:36:54.251 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fe59c700 1 -- 172.21.15.123:0/3401671451 learned_addr learned my addr 172.21.15.123:0/3401671451 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:54.251 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fe59c700 1 -- 172.21.15.123:0/3401671451 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a000fed60 con 0x7f8a00070430 2023-11-15T06:36:54.251 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fe59c700 1 --2- 172.21.15.123:0/3401671451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fe730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f89f000b5f0 tx=0x7f89f0006ee0).ready entity=mon.0 client_cookie=c7f1c1e4021d157d server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fd59a700 1 -- 172.21.15.123:0/3401671451 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f000d600 con 0x7f8a00070430 2023-11-15T06:36:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fd59a700 1 -- 172.21.15.123:0/3401671451 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f89f000dbd0 con 0x7f8a00070430 2023-11-15T06:36:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.246+0000 7f89fd59a700 1 -- 172.21.15.123:0/3401671451 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f0011c00 con 0x7f8a00070430 2023-11-15T06:36:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.247+0000 7f89ff59e700 1 -- 172.21.15.123:0/3401671451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 msgr2=0x7f8a000fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:54.252 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.247+0000 7f89ff59e700 1 --2- 172.21.15.123:0/3401671451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fe730 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f89f000b5f0 tx=0x7f89f0006ee0).stop 2023-11-15T06:36:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.247+0000 7f89ff59e700 1 -- 172.21.15.123:0/3401671451 shutdown_connections 2023-11-15T06:36:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.247+0000 7f89ff59e700 1 --2- 172.21.15.123:0/3401671451 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fe730 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.247+0000 7f89ff59e700 1 -- 172.21.15.123:0/3401671451 >> 172.21.15.123:0/3401671451 conn(0x7f8a000f6210 msgr2=0x7f8a000f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:54.253 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.247+0000 7f89ff59e700 1 -- 172.21.15.123:0/3401671451 shutdown_connections 2023-11-15T06:36:54.254 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.247+0000 7f89ff59e700 1 -- 172.21.15.123:0/3401671451 wait complete. 2023-11-15T06:36:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89ff59e700 1 Processor -- start 2023-11-15T06:36:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89ff59e700 1 -- start start 2023-11-15T06:36:54.256 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89ff59e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fc1b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:54.257 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89ff59e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a000fa630 con 0x7f8a00070430 2023-11-15T06:36:54.257 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89fe59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fc1b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:54.257 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89fe59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fc1b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45958/0 (socket says 172.21.15.123:45958) 2023-11-15T06:36:54.257 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89fe59c700 1 -- 172.21.15.123:0/908047659 learned_addr learned my addr 172.21.15.123:0/908047659 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:54.257 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89fe59c700 1 -- 172.21.15.123:0/908047659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f000b200 con 0x7f8a00070430 2023-11-15T06:36:54.258 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.248+0000 7f89fe59c700 1 --2- 172.21.15.123:0/908047659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fc1b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f89f000b190 tx=0x7f89f0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:54.258 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.249+0000 7f89eeffd700 1 -- 172.21.15.123:0/908047659 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f001b410 con 0x7f8a00070430 2023-11-15T06:36:54.258 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.249+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a000fa7d0 con 0x7f8a00070430 2023-11-15T06:36:54.258 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.249+0000 7f89eeffd700 1 -- 172.21.15.123:0/908047659 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f89f001b9e0 con 0x7f8a00070430 2023-11-15T06:36:54.258 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.249+0000 7f89eeffd700 1 -- 172.21.15.123:0/908047659 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f001a880 con 0x7f8a00070430 2023-11-15T06:36:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.249+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a000fabe0 con 0x7f8a00070430 2023-11-15T06:36:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.249+0000 7f89eeffd700 1 -- 172.21.15.123:0/908047659 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f89f000f070 con 0x7f8a00070430 2023-11-15T06:36:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.249+0000 7f89eeffd700 1 -- 172.21.15.123:0/908047659 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f89f0025c60 con 0x7f8a00070430 2023-11-15T06:36:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.250+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a00045ad0 con 0x7f8a00070430 2023-11-15T06:36:54.259 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.252+0000 7f89eeffd700 1 -- 172.21.15.123:0/908047659 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f89f0016070 con 0x7f8a00070430 2023-11-15T06:36:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.320+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f8a00104670 con 0x7f8a00070430 2023-11-15T06:36:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.321+0000 7f89eeffd700 1 -- 172.21.15.123:0/908047659 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f89f001bb50 con 0x7f8a00070430 2023-11-15T06:36:54.330 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.331 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: { 2023-11-15T06:36:54.331 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "fsid": "4a6dbac0-8381-11ee-8dbe-212e2dc638e7", 2023-11-15T06:36:54.331 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "health": { 2023-11-15T06:36:54.331 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T06:36:54.331 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T06:36:54.332 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T06:36:54.332 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:54.332 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T06:36:54.332 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T06:36:54.332 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 0 2023-11-15T06:36:54.333 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:36:54.333 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T06:36:54.333 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "smithi123" 2023-11-15T06:36:54.333 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:36:54.333 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum_age": 1, 2023-11-15T06:36:54.334 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T06:36:54.334 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:54.334 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T06:36:54.334 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T06:36:54.334 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:54.335 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T06:36:54.335 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:54.335 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T06:36:54.335 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T06:36:54.335 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T06:36:54.336 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T06:36:54.336 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T06:36:54.336 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T06:36:54.336 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:54.336 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T06:36:54.336 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T06:36:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T06:36:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T06:36:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T06:36:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T06:36:54.337 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T06:36:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T06:36:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T06:36:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T06:36:54.338 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T06:36:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T06:36:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T06:36:54.339 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "available": false, 2023-11-15T06:36:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T06:36:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T06:36:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "iostat", 2023-11-15T06:36:54.340 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "restful" 2023-11-15T06:36:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:36:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:36:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T06:36:54.341 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:54.342 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "modified": "2023-11-15T06:36:49.037820+0000", 2023-11-15T06:36:54.342 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:36:54.342 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:54.342 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T06:36:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: } 2023-11-15T06:36:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.322+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 msgr2=0x7f8a000fc1b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.322+0000 7f89ff59e700 1 --2- 172.21.15.123:0/908047659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fc1b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f89f000b190 tx=0x7f89f0006ee0).stop 2023-11-15T06:36:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.322+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 shutdown_connections 2023-11-15T06:36:54.343 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.322+0000 7f89ff59e700 1 --2- 172.21.15.123:0/908047659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a00070430 0x7f8a000fc1b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:54.344 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.322+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 >> 172.21.15.123:0/908047659 conn(0x7f8a000f6210 msgr2=0x7f8a000f7cc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:54.344 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.322+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 shutdown_connections 2023-11-15T06:36:54.344 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:54.322+0000 7f89ff59e700 1 -- 172.21.15.123:0/908047659 wait complete. 2023-11-15T06:36:54.648 INFO:teuthology.orchestra.run.smithi123.stderr:mgr not available, waiting (1/15)... 2023-11-15T06:36:54.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:54 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:54.321904+0000 mon.smithi123 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.123:0/908047659' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T06:36:56.650 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T06:36:57.013 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f9175d08700 1 Processor -- start 2023-11-15T06:36:57.013 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f9175d08700 1 -- start start 2023-11-15T06:36:57.017 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f9175d08700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f91700fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:57.017 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f9175d08700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91700fee10 con 0x7f91700703f0 2023-11-15T06:36:57.017 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f916f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f91700fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f916f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f91700fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45960/0 (socket says 172.21.15.123:45960) 2023-11-15T06:36:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f916f7fe700 1 -- 172.21.15.123:0/2004456736 learned_addr learned my addr 172.21.15.123:0/2004456736 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.010+0000 7f916f7fe700 1 -- 172.21.15.123:0/2004456736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f91700fef50 con 0x7f91700703f0 2023-11-15T06:36:57.018 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.011+0000 7f916f7fe700 1 --2- 172.21.15.123:0/2004456736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f91700fe920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f916000b5f0 tx=0x7f9160006ee0).ready entity=mon.0 client_cookie=a7a104da5a09ae26 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:57.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.011+0000 7f916e7fc700 1 -- 172.21.15.123:0/2004456736 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f916000d600 con 0x7f91700703f0 2023-11-15T06:36:57.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.011+0000 7f916e7fc700 1 -- 172.21.15.123:0/2004456736 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f916000dbd0 con 0x7f91700703f0 2023-11-15T06:36:57.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.011+0000 7f916e7fc700 1 -- 172.21.15.123:0/2004456736 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9160011c00 con 0x7f91700703f0 2023-11-15T06:36:57.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 -- 172.21.15.123:0/2004456736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 msgr2=0x7f91700fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:57.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 --2- 172.21.15.123:0/2004456736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f91700fe920 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f916000b5f0 tx=0x7f9160006ee0).stop 2023-11-15T06:36:57.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 -- 172.21.15.123:0/2004456736 shutdown_connections 2023-11-15T06:36:57.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 --2- 172.21.15.123:0/2004456736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f91700fe920 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:57.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 -- 172.21.15.123:0/2004456736 >> 172.21.15.123:0/2004456736 conn(0x7f91700f63c0 msgr2=0x7f91700f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:57.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 -- 172.21.15.123:0/2004456736 shutdown_connections 2023-11-15T06:36:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 -- 172.21.15.123:0/2004456736 wait complete. 2023-11-15T06:36:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.012+0000 7f9175d08700 1 Processor -- start 2023-11-15T06:36:57.021 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.013+0000 7f9175d08700 1 -- start start 2023-11-15T06:36:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.013+0000 7f9175d08700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f917010a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.013+0000 7f9175d08700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f917010aeb0 con 0x7f91700703f0 2023-11-15T06:36:57.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.013+0000 7f916f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f917010a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:57.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.013+0000 7f916f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f917010a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:45972/0 (socket says 172.21.15.123:45972) 2023-11-15T06:36:57.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.013+0000 7f916f7fe700 1 -- 172.21.15.123:0/2690389310 learned_addr learned my addr 172.21.15.123:0/2690389310 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:57.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.013+0000 7f916f7fe700 1 -- 172.21.15.123:0/2690389310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f916000b200 con 0x7f91700703f0 2023-11-15T06:36:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f916f7fe700 1 --2- 172.21.15.123:0/2690389310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f917010a9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f916000b190 tx=0x7f9160006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:57.024 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f9157fff700 1 -- 172.21.15.123:0/2690389310 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f916001b410 con 0x7f91700703f0 2023-11-15T06:36:57.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f9157fff700 1 -- 172.21.15.123:0/2690389310 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f916001b9e0 con 0x7f91700703f0 2023-11-15T06:36:57.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f917010b0b0 con 0x7f91700703f0 2023-11-15T06:36:57.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f9157fff700 1 -- 172.21.15.123:0/2690389310 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f916001a880 con 0x7f91700703f0 2023-11-15T06:36:57.026 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f917010b4c0 con 0x7f91700703f0 2023-11-15T06:36:57.026 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f9157fff700 1 -- 172.21.15.123:0/2690389310 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f916000f070 con 0x7f91700703f0 2023-11-15T06:36:57.026 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.014+0000 7f9157fff700 1 -- 172.21.15.123:0/2690389310 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9160025b70 con 0x7f91700703f0 2023-11-15T06:36:57.027 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.015+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9170045ad0 con 0x7f91700703f0 2023-11-15T06:36:57.027 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.017+0000 7f9157fff700 1 -- 172.21.15.123:0/2690389310 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9160016070 con 0x7f91700703f0 2023-11-15T06:36:57.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.085+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f91701047f0 con 0x7f91700703f0 2023-11-15T06:36:57.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.086+0000 7f9157fff700 1 -- 172.21.15.123:0/2690389310 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f9160006a70 con 0x7f91700703f0 2023-11-15T06:36:57.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: { 2023-11-15T06:36:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "fsid": "4a6dbac0-8381-11ee-8dbe-212e2dc638e7", 2023-11-15T06:36:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "health": { 2023-11-15T06:36:57.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T06:36:57.096 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T06:36:57.096 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T06:36:57.096 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:57.096 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T06:36:57.096 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T06:36:57.097 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 0 2023-11-15T06:36:57.097 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:36:57.097 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T06:36:57.097 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "smithi123" 2023-11-15T06:36:57.097 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:36:57.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum_age": 4, 2023-11-15T06:36:57.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T06:36:57.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:57.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T06:36:57.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T06:36:57.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:57.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T06:36:57.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:57.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T06:36:57.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T06:36:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T06:36:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T06:36:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T06:36:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T06:36:57.100 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:57.101 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T06:36:57.101 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T06:36:57.101 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T06:36:57.101 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T06:36:57.101 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T06:36:57.102 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T06:36:57.102 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T06:36:57.102 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T06:36:57.102 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T06:36:57.103 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:57.103 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T06:36:57.103 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:57.103 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T06:36:57.103 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T06:36:57.104 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:57.104 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T06:36:57.104 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "available": false, 2023-11-15T06:36:57.104 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T06:36:57.104 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T06:36:57.105 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "iostat", 2023-11-15T06:36:57.105 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "restful" 2023-11-15T06:36:57.105 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:36:57.105 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:36:57.105 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:57.105 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T06:36:57.106 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:36:57.106 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "modified": "2023-11-15T06:36:49.037820+0000", 2023-11-15T06:36:57.106 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:36:57.106 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:36:57.106 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T06:36:57.107 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: } 2023-11-15T06:36:57.107 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.087+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 msgr2=0x7f917010a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:57.107 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.087+0000 7f9175d08700 1 --2- 172.21.15.123:0/2690389310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f917010a9c0 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f916000b190 tx=0x7f9160006ee0).stop 2023-11-15T06:36:57.107 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.087+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 shutdown_connections 2023-11-15T06:36:57.107 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.087+0000 7f9175d08700 1 --2- 172.21.15.123:0/2690389310 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91700703f0 0x7f917010a9c0 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:57.108 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.087+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 >> 172.21.15.123:0/2690389310 conn(0x7f91700f63c0 msgr2=0x7f91700f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:57.108 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.087+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 shutdown_connections 2023-11-15T06:36:57.108 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:57.087+0000 7f9175d08700 1 -- 172.21.15.123:0/2690389310 wait complete. 2023-11-15T06:36:57.392 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:57.086802+0000 mon.smithi123 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.123:0/2690389310' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T06:36:57.449 INFO:teuthology.orchestra.run.smithi123.stderr:mgr not available, waiting (2/15)... 2023-11-15T06:36:59.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:58.743590+0000 mon.smithi123 (mon.0) 9 : cluster [INF] Activating manager daemon smithi123.razjtn 2023-11-15T06:36:59.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:58.746173+0000 mon.smithi123 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi123.razjtn(active, starting, since 0.00268574s) 2023-11-15T06:36:59.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.746805+0000 mon.smithi123 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:36:59.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.746988+0000 mon.smithi123 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:36:59.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.747147+0000 mon.smithi123 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:36:59.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.747313+0000 mon.smithi123 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2023-11-15T06:36:59.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.747731+0000 mon.smithi123 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mgr metadata", "who": "smithi123.razjtn", "id": "smithi123.razjtn"}]: dispatch 2023-11-15T06:36:59.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:58.761135+0000 mon.smithi123 (mon.0) 16 : cluster [INF] Manager daemon smithi123.razjtn is now available 2023-11-15T06:36:59.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.769067+0000 mon.smithi123 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' 2023-11-15T06:36:59.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.772628+0000 mon.smithi123 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:36:59.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.773698+0000 mon.smithi123 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:36:59.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.777853+0000 mon.smithi123 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' 2023-11-15T06:36:59.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:36:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:36:58.780041+0000 mon.smithi123 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.123:0/2110852386' entity='mgr.smithi123.razjtn' 2023-11-15T06:36:59.451 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2023-11-15T06:36:59.801 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef9fb4700 1 Processor -- start 2023-11-15T06:36:59.801 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef9fb4700 1 -- start start 2023-11-15T06:36:59.801 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef9fb4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:59.802 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef9fb4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef40fc5c0 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.802 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:59.802 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35034/0 (socket says 172.21.15.123:35034) 2023-11-15T06:36:59.803 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef37fe700 1 -- 172.21.15.123:0/4102804284 learned_addr learned my addr 172.21.15.123:0/4102804284 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:59.803 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef37fe700 1 -- 172.21.15.123:0/4102804284 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ef40fc700 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.803 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef37fe700 1 --2- 172.21.15.123:0/4102804284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5edc00b5f0 tx=0x7f5edc006ee0).ready entity=mon.0 client_cookie=a0177026c52cd008 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:59.803 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef27fc700 1 -- 172.21.15.123:0/4102804284 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5edc00d600 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.803 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.795+0000 7f5ef27fc700 1 -- 172.21.15.123:0/4102804284 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5edc00dbd0 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.804 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef27fc700 1 -- 172.21.15.123:0/4102804284 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5edc011c00 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.804 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/4102804284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 msgr2=0x7f5ef40fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:36:59.804 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef9fb4700 1 --2- 172.21.15.123:0/4102804284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40fc0d0 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f5edc00b5f0 tx=0x7f5edc006ee0).stop 2023-11-15T06:36:59.804 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/4102804284 shutdown_connections 2023-11-15T06:36:59.805 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef9fb4700 1 --2- 172.21.15.123:0/4102804284 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40fc0d0 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:36:59.805 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/4102804284 >> 172.21.15.123:0/4102804284 conn(0x7f5ef40f74b0 msgr2=0x7f5ef40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:36:59.805 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/4102804284 shutdown_connections 2023-11-15T06:36:59.805 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.796+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/4102804284 wait complete. 2023-11-15T06:36:59.805 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef9fb4700 1 Processor -- start 2023-11-15T06:36:59.806 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef9fb4700 1 -- start start 2023-11-15T06:36:59.806 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef9fb4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:59.806 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef9fb4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ef4070a60 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.807 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:59.807 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35040/0 (socket says 172.21.15.123:35040) 2023-11-15T06:36:59.807 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef37fe700 1 -- 172.21.15.123:0/3513897662 learned_addr learned my addr 172.21.15.123:0/3513897662 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:36:59.807 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef37fe700 1 -- 172.21.15.123:0/3513897662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5edc00b200 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.808 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.797+0000 7f5ef37fe700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40725e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f5edc00b1d0 tx=0x7f5edc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:59.808 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.798+0000 7f5eebfff700 1 -- 172.21.15.123:0/3513897662 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5edc01b410 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.808 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.798+0000 7f5eebfff700 1 -- 172.21.15.123:0/3513897662 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5edc01b9e0 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.798+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ef4070c00 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.798+0000 7f5eebfff700 1 -- 172.21.15.123:0/3513897662 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5edc01a840 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.809 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.798+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ef4071010 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.798+0000 7f5eebfff700 1 -- 172.21.15.123:0/3513897662 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7f5edc00f070 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.798+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ef4045ad0 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.810 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.799+0000 7f5eebfff700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5ed402fc50 0x7f5ed4032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:36:59.811 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.799+0000 7f5eebfff700 1 -- 172.21.15.123:0/3513897662 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5edc0412a0 con 0x7f5ef40fbcb0 2023-11-15T06:36:59.812 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.799+0000 7f5ef2ffd700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5ed402fc50 0x7f5ed4032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:36:59.812 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.799+0000 7f5ef2ffd700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5ed402fc50 0x7f5ed4032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5ee40096a0 tx=0x7f5ee4006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:36:59.812 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:36:59.803+0000 7f5eebfff700 1 -- 172.21.15.123:0/3513897662 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5edc022b70 con 0x7f5ef40fbcb0 2023-11-15T06:37:00.015 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.006+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5ef40fdad0 con 0x7f5ef40fbcb0 2023-11-15T06:37:00.015 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.006+0000 7f5eebfff700 1 -- 172.21.15.123:0/3513897662 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7f5edc016030 con 0x7f5ef40fbcb0 2023-11-15T06:37:00.016 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.016 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: { 2023-11-15T06:37:00.017 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "fsid": "4a6dbac0-8381-11ee-8dbe-212e2dc638e7", 2023-11-15T06:37:00.017 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "health": { 2023-11-15T06:37:00.017 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2023-11-15T06:37:00.018 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "checks": {}, 2023-11-15T06:37:00.018 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mutes": [] 2023-11-15T06:37:00.018 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:37:00.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "election_epoch": 5, 2023-11-15T06:37:00.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum": [ 2023-11-15T06:37:00.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 0 2023-11-15T06:37:00.019 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:37:00.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum_names": [ 2023-11-15T06:37:00.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "smithi123" 2023-11-15T06:37:00.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:37:00.020 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "quorum_age": 7, 2023-11-15T06:37:00.021 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "monmap": { 2023-11-15T06:37:00.021 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:37:00.021 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2023-11-15T06:37:00.021 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_mons": 1 2023-11-15T06:37:00.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:37:00.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osdmap": { 2023-11-15T06:37:00.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:37:00.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_osds": 0, 2023-11-15T06:37:00.022 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_up_osds": 0, 2023-11-15T06:37:00.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osd_up_since": 0, 2023-11-15T06:37:00.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_in_osds": 0, 2023-11-15T06:37:00.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "osd_in_since": 0, 2023-11-15T06:37:00.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2023-11-15T06:37:00.023 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:37:00.024 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "pgmap": { 2023-11-15T06:37:00.024 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "pgs_by_state": [], 2023-11-15T06:37:00.024 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_pgs": 0, 2023-11-15T06:37:00.024 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_pools": 0, 2023-11-15T06:37:00.024 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_objects": 0, 2023-11-15T06:37:00.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "data_bytes": 0, 2023-11-15T06:37:00.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_used": 0, 2023-11-15T06:37:00.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_avail": 0, 2023-11-15T06:37:00.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "bytes_total": 0 2023-11-15T06:37:00.025 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:37:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "fsmap": { 2023-11-15T06:37:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:37:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "by_rank": [], 2023-11-15T06:37:00.026 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "up:standby": 0 2023-11-15T06:37:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:37:00.027 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mgrmap": { 2023-11-15T06:37:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "available": true, 2023-11-15T06:37:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_standbys": 0, 2023-11-15T06:37:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "modules": [ 2023-11-15T06:37:00.028 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "iostat", 2023-11-15T06:37:00.030 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "restful" 2023-11-15T06:37:00.030 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ], 2023-11-15T06:37:00.030 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:37:00.030 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:37:00.030 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "servicemap": { 2023-11-15T06:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 1, 2023-11-15T06:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "modified": "2023-11-15T06:36:49.037820+0000", 2023-11-15T06:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "services": {} 2023-11-15T06:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: }, 2023-11-15T06:37:00.031 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "progress_events": {} 2023-11-15T06:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: } 2023-11-15T06:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5ed402fc50 msgr2=0x7f5ed4032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5ed402fc50 0x7f5ed4032110 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f5ee40096a0 tx=0x7f5ee4006b40).stop 2023-11-15T06:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 msgr2=0x7f5ef40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:00.032 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40725e0 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f5edc00b1d0 tx=0x7f5edc006ee0).stop 2023-11-15T06:37:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 shutdown_connections 2023-11-15T06:37:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5ed402fc50 0x7f5ed4032110 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 --2- 172.21.15.123:0/3513897662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5ef40fbcb0 0x7f5ef40725e0 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 >> 172.21.15.123:0/3513897662 conn(0x7f5ef40f74b0 msgr2=0x7f5ef40f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 shutdown_connections 2023-11-15T06:37:00.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.009+0000 7f5ef9fb4700 1 -- 172.21.15.123:0/3513897662 wait complete. 2023-11-15T06:37:00.373 INFO:teuthology.orchestra.run.smithi123.stderr:mgr is available 2023-11-15T06:37:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:Enabling cephadm module... 2023-11-15T06:37:00.374 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2023-11-15T06:37:00.730 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.724+0000 7fde176d8700 1 Processor -- start 2023-11-15T06:37:00.730 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.724+0000 7fde176d8700 1 -- start start 2023-11-15T06:37:00.731 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.724+0000 7fde176d8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:00.731 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.724+0000 7fde176d8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde100fedd0 con 0x7fde10070430 2023-11-15T06:37:00.731 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fde15474700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:00.731 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fde15474700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35048/0 (socket says 172.21.15.123:35048) 2023-11-15T06:37:00.732 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fde15474700 1 -- 172.21.15.123:0/2781024160 learned_addr learned my addr 172.21.15.123:0/2781024160 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:00.732 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fde15474700 1 -- 172.21.15.123:0/2781024160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde100fef10 con 0x7fde10070430 2023-11-15T06:37:00.732 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fde15474700 1 --2- 172.21.15.123:0/2781024160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fde0000b5f0 tx=0x7fde00006ee0).ready entity=mon.0 client_cookie=2e6be10c1f58e462 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:00.732 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fddfffff700 1 -- 172.21.15.123:0/2781024160 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0000d600 con 0x7fde10070430 2023-11-15T06:37:00.732 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fddfffff700 1 -- 172.21.15.123:0/2781024160 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fde0000dbd0 con 0x7fde10070430 2023-11-15T06:37:00.733 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.725+0000 7fddfffff700 1 -- 172.21.15.123:0/2781024160 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde00011c00 con 0x7fde10070430 2023-11-15T06:37:00.733 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 -- 172.21.15.123:0/2781024160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 msgr2=0x7fde100fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:00.733 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 --2- 172.21.15.123:0/2781024160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fe8e0 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fde0000b5f0 tx=0x7fde00006ee0).stop 2023-11-15T06:37:00.733 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 -- 172.21.15.123:0/2781024160 shutdown_connections 2023-11-15T06:37:00.733 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 --2- 172.21.15.123:0/2781024160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fe8e0 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:00.734 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 -- 172.21.15.123:0/2781024160 >> 172.21.15.123:0/2781024160 conn(0x7fde100f63c0 msgr2=0x7fde100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:00.734 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 -- 172.21.15.123:0/2781024160 shutdown_connections 2023-11-15T06:37:00.734 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 -- 172.21.15.123:0/2781024160 wait complete. 2023-11-15T06:37:00.734 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 Processor -- start 2023-11-15T06:37:00.734 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.726+0000 7fde176d8700 1 -- start start 2023-11-15T06:37:00.735 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde176d8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:00.735 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde176d8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde100fa7e0 con 0x7fde10070430 2023-11-15T06:37:00.735 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde15474700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:00.735 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde15474700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35060/0 (socket says 172.21.15.123:35060) 2023-11-15T06:37:00.736 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde15474700 1 -- 172.21.15.123:0/2442361279 learned_addr learned my addr 172.21.15.123:0/2442361279 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:00.736 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde15474700 1 -- 172.21.15.123:0/2442361279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde0000b200 con 0x7fde10070430 2023-11-15T06:37:00.736 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde15474700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fc360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fde0000b190 tx=0x7fde00006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:00.736 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0001b410 con 0x7fde10070430 2023-11-15T06:37:00.736 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde100fa980 con 0x7fde10070430 2023-11-15T06:37:00.737 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fde0001b9e0 con 0x7fde10070430 2023-11-15T06:37:00.737 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde0001a880 con 0x7fde10070430 2023-11-15T06:37:00.737 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.727+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde100fad90 con 0x7fde10070430 2023-11-15T06:37:00.738 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.728+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7fde0000f070 con 0x7fde10070430 2023-11-15T06:37:00.738 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.729+0000 7fddfdffb700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7fddf802fc50 0x7fddf8032110 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:00.738 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.729+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fde00041070 con 0x7fde10070430 2023-11-15T06:37:00.739 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.729+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde10045ad0 con 0x7fde10070430 2023-11-15T06:37:00.739 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.729+0000 7fde14c73700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7fddf802fc50 0x7fddf8032110 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:00.739 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.729+0000 7fde14c73700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7fddf802fc50 0x7fddf8032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fde0c0096a0 tx=0x7fde0c006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:00.739 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.733+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde00016070 con 0x7fde10070430 2023-11-15T06:37:00.754 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.751+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7fde0001b570 con 0x7fde10070430 2023-11-15T06:37:00.922 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:00.919+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7fde10027a00 con 0x7fde10070430 2023-11-15T06:37:01.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:00 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:36:59.749789+0000 mon.smithi123 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi123.razjtn(active, since 1.0063s) 2023-11-15T06:37:01.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:00 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:00.007247+0000 mon.smithi123 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.123:0/3513897662' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2023-11-15T06:37:01.755 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.752+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fde00019300 con 0x7fde10070430 2023-11-15T06:37:01.756 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.754+0000 7fddfdffb700 1 -- 172.21.15.123:0/2442361279 <== mon.0 v2:172.21.15.123:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7fde00022520 con 0x7fde10070430 2023-11-15T06:37:01.759 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7fddf802fc50 msgr2=0x7fddf8032110 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7fddf802fc50 0x7fddf8032110 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fde0c0096a0 tx=0x7fde0c006b40).stop 2023-11-15T06:37:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 msgr2=0x7fde100fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fc360 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7fde0000b190 tx=0x7fde00006ee0).stop 2023-11-15T06:37:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 shutdown_connections 2023-11-15T06:37:01.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7fddf802fc50 0x7fddf8032110 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:01.761 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 --2- 172.21.15.123:0/2442361279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fde10070430 0x7fde100fc360 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:01.761 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.756+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 >> 172.21.15.123:0/2442361279 conn(0x7fde100f63c0 msgr2=0x7fde100f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:01.761 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.757+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 shutdown_connections 2023-11-15T06:37:01.762 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:01.757+0000 7fde176d8700 1 -- 172.21.15.123:0/2442361279 wait complete. 2023-11-15T06:37:02.066 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:00.754051+0000 mon.smithi123 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi123.razjtn(active, since 2s) 2023-11-15T06:37:02.067 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:00.920925+0000 mon.smithi123 (mon.0) 25 : audit [INF] from='client.? 172.21.15.123:0/2442361279' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2023-11-15T06:37:02.122 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T06:37:02.489 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.485+0000 7f0dfa63a700 1 Processor -- start 2023-11-15T06:37:02.489 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.485+0000 7f0dfa63a700 1 -- start start 2023-11-15T06:37:02.489 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.485+0000 7f0dfa63a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df40fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:02.489 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.485+0000 7f0dfa63a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df40fee30 con 0x7f0df40703d0 2023-11-15T06:37:02.489 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.485+0000 7f0df3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df40fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:02.490 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.485+0000 7f0df3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df40fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35104/0 (socket says 172.21.15.123:35104) 2023-11-15T06:37:02.490 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.485+0000 7f0df3fff700 1 -- 172.21.15.123:0/1802235894 learned_addr learned my addr 172.21.15.123:0/1802235894 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:02.490 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.486+0000 7f0df3fff700 1 -- 172.21.15.123:0/1802235894 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0df40fef70 con 0x7f0df40703d0 2023-11-15T06:37:02.490 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.486+0000 7f0df3fff700 1 --2- 172.21.15.123:0/1802235894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df40fe940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0de400b5f0 tx=0x7f0de4006ee0).ready entity=mon.0 client_cookie=6d2b8875d56b8c56 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:02.490 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.486+0000 7f0df2ffd700 1 -- 172.21.15.123:0/1802235894 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de400d600 con 0x7f0df40703d0 2023-11-15T06:37:02.491 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.486+0000 7f0df2ffd700 1 -- 172.21.15.123:0/1802235894 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0de400dbd0 con 0x7f0df40703d0 2023-11-15T06:37:02.491 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.486+0000 7f0df2ffd700 1 -- 172.21.15.123:0/1802235894 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de4011c00 con 0x7f0df40703d0 2023-11-15T06:37:02.493 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.487+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1802235894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 msgr2=0x7f0df40fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:02.493 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.487+0000 7f0dfa63a700 1 --2- 172.21.15.123:0/1802235894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df40fe940 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f0de400b5f0 tx=0x7f0de4006ee0).stop 2023-11-15T06:37:02.494 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.487+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1802235894 shutdown_connections 2023-11-15T06:37:02.494 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.487+0000 7f0dfa63a700 1 --2- 172.21.15.123:0/1802235894 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df40fe940 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:02.494 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.487+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1802235894 >> 172.21.15.123:0/1802235894 conn(0x7f0df40f63c0 msgr2=0x7f0df40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:02.494 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.487+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1802235894 shutdown_connections 2023-11-15T06:37:02.494 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.487+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1802235894 wait complete. 2023-11-15T06:37:02.495 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.488+0000 7f0dfa63a700 1 Processor -- start 2023-11-15T06:37:02.495 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.488+0000 7f0dfa63a700 1 -- start start 2023-11-15T06:37:02.495 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.488+0000 7f0dfa63a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df410aa30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:02.495 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.488+0000 7f0dfa63a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0df410af20 con 0x7f0df40703d0 2023-11-15T06:37:02.495 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.488+0000 7f0df3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df410aa30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:02.496 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.488+0000 7f0df3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df410aa30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35108/0 (socket says 172.21.15.123:35108) 2023-11-15T06:37:02.496 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.488+0000 7f0df3fff700 1 -- 172.21.15.123:0/1768864723 learned_addr learned my addr 172.21.15.123:0/1768864723 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:02.496 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.489+0000 7f0df3fff700 1 -- 172.21.15.123:0/1768864723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0de400b200 con 0x7f0df40703d0 2023-11-15T06:37:02.496 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.489+0000 7f0df3fff700 1 --2- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df410aa30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0de400e040 tx=0x7f0de4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:02.496 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.489+0000 7f0df0ff9700 1 -- 172.21.15.123:0/1768864723 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de400d9f0 con 0x7f0df40703d0 2023-11-15T06:37:02.496 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.489+0000 7f0df0ff9700 1 -- 172.21.15.123:0/1768864723 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0de401a9f0 con 0x7f0df40703d0 2023-11-15T06:37:02.497 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.489+0000 7f0df0ff9700 1 -- 172.21.15.123:0/1768864723 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0de401b8c0 con 0x7f0df40703d0 2023-11-15T06:37:02.497 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.489+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0df410b120 con 0x7f0df40703d0 2023-11-15T06:37:02.497 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.489+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0df410b530 con 0x7f0df40703d0 2023-11-15T06:37:02.497 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.490+0000 7f0df0ff9700 1 -- 172.21.15.123:0/1768864723 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0de400f070 con 0x7f0df40703d0 2023-11-15T06:37:02.497 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.490+0000 7f0df0ff9700 1 --2- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f0de002fdb0 0x7f0de0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:02.498 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.490+0000 7f0df0ff9700 1 -- 172.21.15.123:0/1768864723 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f0de40420e0 con 0x7f0df40703d0 2023-11-15T06:37:02.498 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.490+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0df4045ad0 con 0x7f0df40703d0 2023-11-15T06:37:02.498 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.490+0000 7f0df37fe700 1 -- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f0de002fdb0 msgr2=0x7f0de0032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/982215078 2023-11-15T06:37:02.498 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.490+0000 7f0df37fe700 1 --2- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f0de002fdb0 0x7f0de0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:37:02.499 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.494+0000 7f0df0ff9700 1 -- 172.21.15.123:0/1768864723 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0de4016070 con 0x7f0df40703d0 2023-11-15T06:37:02.689 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.686+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f0df4104890 con 0x7f0df40703d0 2023-11-15T06:37:02.690 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.687+0000 7f0df0ff9700 1 -- 172.21.15.123:0/1768864723 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7f0de4025460 con 0x7f0df40703d0 2023-11-15T06:37:02.690 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: { 2023-11-15T06:37:02.690 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 5, 2023-11-15T06:37:02.690 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "available": true, 2023-11-15T06:37:02.691 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "active_name": "smithi123.razjtn", 2023-11-15T06:37:02.691 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T06:37:02.691 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: } 2023-11-15T06:37:02.692 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f0de002fdb0 msgr2=0x7f0de0032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:37:02.692 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 --2- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f0de002fdb0 0x7f0de0032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:02.692 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 msgr2=0x7f0df410aa30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:02.692 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 --2- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df410aa30 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f0de400e040 tx=0x7f0de4006ee0).stop 2023-11-15T06:37:02.693 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 shutdown_connections 2023-11-15T06:37:02.694 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 --2- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f0de002fdb0 0x7f0de0032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:02.694 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 --2- 172.21.15.123:0/1768864723 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0df40703d0 0x7f0df410aa30 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:02.694 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 >> 172.21.15.123:0/1768864723 conn(0x7f0df40f63c0 msgr2=0x7f0df40f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:02.694 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.689+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 shutdown_connections 2023-11-15T06:37:02.694 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:02.690+0000 7f0dfa63a700 1 -- 172.21.15.123:0/1768864723 wait complete. 2023-11-15T06:37:02.993 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:02 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:01.754920+0000 mon.smithi123 (mon.0) 26 : audit [INF] from='client.? 172.21.15.123:0/2442361279' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2023-11-15T06:37:02.993 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:02 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:01.754993+0000 mon.smithi123 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi123.razjtn(active, since 3s) 2023-11-15T06:37:02.994 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:02 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:02.687628+0000 mon.smithi123 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.123:0/1768864723' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T06:37:03.033 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for the mgr to restart... 2023-11-15T06:37:03.033 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for mgr epoch 5... 2023-11-15T06:37:03.033 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T06:37:03.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.423+0000 7f5d8f77c700 1 Processor -- start 2023-11-15T06:37:03.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.423+0000 7f5d8f77c700 1 -- start start 2023-11-15T06:37:03.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.423+0000 7f5d8f77c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d880fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.423+0000 7f5d8f77c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d880fb490 con 0x7f5d880fab80 2023-11-15T06:37:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d8d518700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d880fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d8d518700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d880fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35118/0 (socket says 172.21.15.123:35118) 2023-11-15T06:37:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d8d518700 1 -- 172.21.15.123:0/1649837365 learned_addr learned my addr 172.21.15.123:0/1649837365 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:03.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d8d518700 1 -- 172.21.15.123:0/1649837365 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d880fb5d0 con 0x7f5d880fab80 2023-11-15T06:37:03.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d8d518700 1 --2- 172.21.15.123:0/1649837365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d880fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5d8400b5f0 tx=0x7f5d84006ee0).ready entity=mon.0 client_cookie=b2133fba532f9f43 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:03.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d77fff700 1 -- 172.21.15.123:0/1649837365 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d8400d600 con 0x7f5d880fab80 2023-11-15T06:37:03.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d77fff700 1 -- 172.21.15.123:0/1649837365 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5d8400dbd0 con 0x7f5d880fab80 2023-11-15T06:37:03.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.424+0000 7f5d77fff700 1 -- 172.21.15.123:0/1649837365 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d84011c00 con 0x7f5d880fab80 2023-11-15T06:37:03.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.425+0000 7f5d8f77c700 1 -- 172.21.15.123:0/1649837365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 msgr2=0x7f5d880fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:03.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.425+0000 7f5d8f77c700 1 --2- 172.21.15.123:0/1649837365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d880fafa0 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f5d8400b5f0 tx=0x7f5d84006ee0).stop 2023-11-15T06:37:03.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.425+0000 7f5d8f77c700 1 -- 172.21.15.123:0/1649837365 shutdown_connections 2023-11-15T06:37:03.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.425+0000 7f5d8f77c700 1 --2- 172.21.15.123:0/1649837365 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d880fafa0 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:03.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.425+0000 7f5d8f77c700 1 -- 172.21.15.123:0/1649837365 >> 172.21.15.123:0/1649837365 conn(0x7f5d880f63c0 msgr2=0x7f5d880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:03.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.425+0000 7f5d8f77c700 1 -- 172.21.15.123:0/1649837365 shutdown_connections 2023-11-15T06:37:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.425+0000 7f5d8f77c700 1 -- 172.21.15.123:0/1649837365 wait complete. 2023-11-15T06:37:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8f77c700 1 Processor -- start 2023-11-15T06:37:03.432 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8f77c700 1 -- start start 2023-11-15T06:37:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8f77c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d8810ccc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8f77c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5d8810d1b0 con 0x7f5d880fab80 2023-11-15T06:37:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8d518700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d8810ccc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:03.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8d518700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d8810ccc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:35122/0 (socket says 172.21.15.123:35122) 2023-11-15T06:37:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8d518700 1 -- 172.21.15.123:0/3137457259 learned_addr learned my addr 172.21.15.123:0/3137457259 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.426+0000 7f5d8d518700 1 -- 172.21.15.123:0/3137457259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5d8400b200 con 0x7f5d880fab80 2023-11-15T06:37:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.427+0000 7f5d8d518700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d8810ccc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5d8400e040 tx=0x7f5d84006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.427+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d8401b410 con 0x7f5d880fab80 2023-11-15T06:37:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.427+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5d8810d3b0 con 0x7f5d880fab80 2023-11-15T06:37:03.435 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.427+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f5d8401b9e0 con 0x7f5d880fab80 2023-11-15T06:37:03.436 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.427+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5d8401a820 con 0x7f5d880fab80 2023-11-15T06:37:03.436 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.427+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5d8810d7c0 con 0x7f5d880fab80 2023-11-15T06:37:03.436 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.428+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f5d8400f070 con 0x7f5d880fab80 2023-11-15T06:37:03.436 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.428+0000 7f5d75ffb700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:03.436 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.428+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 --> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d700328c0 con 0x7f5d7002fd40 2023-11-15T06:37:03.437 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.428+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5d84041380 con 0x7f5d880fab80 2023-11-15T06:37:03.437 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.428+0000 7f5d8cd17700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 msgr2=0x7f5d70032200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/982215078 2023-11-15T06:37:03.437 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.428+0000 7f5d8cd17700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:37:03.631 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.629+0000 7f5d8cd17700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 msgr2=0x7f5d70032200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/982215078 2023-11-15T06:37:03.632 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:03.629+0000 7f5d8cd17700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:37:04.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:04.030+0000 7f5d8cd17700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 msgr2=0x7f5d70032200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/982215078 2023-11-15T06:37:04.033 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:04.030+0000 7f5d8cd17700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T06:37:04.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:04.831+0000 7f5d8cd17700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 msgr2=0x7f5d70032200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/982215078 2023-11-15T06:37:04.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:04.831+0000 7f5d8cd17700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T06:37:06.435 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:06.432+0000 7f5d8cd17700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 msgr2=0x7f5d70032200 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/982215078 2023-11-15T06:37:06.436 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:06.432+0000 7f5d8cd17700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T06:37:06.738 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:06.735+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f5d8401bb50 con 0x7f5d880fab80 2023-11-15T06:37:06.738 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:06.735+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 msgr2=0x7f5d70032200 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:37:06.738 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:06.735+0000 7f5d75ffb700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:07.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:06 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:06.681569+0000 mon.smithi123 (mon.0) 29 : cluster [INF] Active manager daemon smithi123.razjtn restarted 2023-11-15T06:37:07.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:06 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:06.681931+0000 mon.smithi123 (mon.0) 30 : cluster [INF] Activating manager daemon smithi123.razjtn 2023-11-15T06:37:07.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:06 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:06.686089+0000 mon.smithi123 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2023-11-15T06:37:07.745 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.741+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f5d84042cc0 con 0x7f5d880fab80 2023-11-15T06:37:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.742+0000 7f5d75ffb700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f5d70033810 0x7f5d70035c00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:07.746 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.742+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f5d700328c0 con 0x7f5d70033810 2023-11-15T06:37:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.742+0000 7f5d8cd17700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f5d70033810 0x7f5d70035c00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.742+0000 7f5d8cd17700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f5d70033810 0x7f5d70035c00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5d78006fd0 tx=0x7f5d78006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:07.747 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.743+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f5d700328c0 con 0x7f5d70033810 2023-11-15T06:37:07.756 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.750+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f5d88059100 con 0x7f5d70033810 2023-11-15T06:37:07.756 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d75ffb700 1 -- 172.21.15.123:0/3137457259 <== mgr.14114 v2:172.21.15.123:6800/4164871557 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f5d88059100 con 0x7f5d70033810 2023-11-15T06:37:07.757 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: { 2023-11-15T06:37:07.757 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2023-11-15T06:37:07.757 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T06:37:07.758 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: } 2023-11-15T06:37:07.758 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f5d70033810 msgr2=0x7f5d70035c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:07.758 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f5d70033810 0x7f5d70035c00 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f5d78006fd0 tx=0x7f5d78006cf0).stop 2023-11-15T06:37:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 msgr2=0x7f5d8810ccc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d8810ccc0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f5d8400e040 tx=0x7f5d84006ee0).stop 2023-11-15T06:37:07.759 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 shutdown_connections 2023-11-15T06:37:07.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f5d70033810 0x7f5d70035c00 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:07.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5d880fab80 0x7f5d8810ccc0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:07.760 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 --2- 172.21.15.123:0/3137457259 >> [v2:172.21.15.123:6800/982215078,v1:172.21.15.123:6801/982215078] conn(0x7f5d7002fd40 0x7f5d70032200 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:07.761 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 >> 172.21.15.123:0/3137457259 conn(0x7f5d880f63c0 msgr2=0x7f5d880ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:07.761 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 shutdown_connections 2023-11-15T06:37:07.762 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:07.751+0000 7f5d8f77c700 1 -- 172.21.15.123:0/3137457259 wait complete. 2023-11-15T06:37:08.091 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:06.736813+0000 mon.smithi123 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi123.razjtn(active, starting, since 0.0549516s) 2023-11-15T06:37:08.091 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.738490+0000 mon.smithi123 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2023-11-15T06:37:08.091 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.739107+0000 mon.smithi123 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mgr metadata", "who": "smithi123.razjtn", "id": "smithi123.razjtn"}]: dispatch 2023-11-15T06:37:08.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.739923+0000 mon.smithi123 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:37:08.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.741610+0000 mon.smithi123 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:37:08.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.741770+0000 mon.smithi123 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:37:08.092 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:06.755403+0000 mon.smithi123 (mon.0) 38 : cluster [INF] Manager daemon smithi123.razjtn is now available 2023-11-15T06:37:08.093 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.768835+0000 mon.smithi123 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:08.093 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.771418+0000 mon.smithi123 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:08.093 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.773878+0000 mon.smithi123 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:08.093 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.860938+0000 mon.smithi123 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:08.093 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.861608+0000 mon.smithi123 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:08.094 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.862685+0000 mon.smithi123 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:08.094 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.863958+0000 mon.smithi123 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:08.094 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.874358+0000 mon.smithi123 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:37:08.094 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:06.875444+0000 mon.smithi123 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:37:08.139 INFO:teuthology.orchestra.run.smithi123.stderr:mgr epoch 5 is available 2023-11-15T06:37:08.139 INFO:teuthology.orchestra.run.smithi123.stderr:Setting orchestrator backend to cephadm... 2023-11-15T06:37:08.139 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2023-11-15T06:37:08.504 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.500+0000 7fafb5daf700 1 Processor -- start 2023-11-15T06:37:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.500+0000 7fafb5daf700 1 -- start start 2023-11-15T06:37:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.500+0000 7fafb5daf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb00fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.500+0000 7fafb5daf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb0070790 con 0x7fafb0070370 2023-11-15T06:37:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.500+0000 7fafaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb00fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:08.505 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.500+0000 7fafaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb00fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44488/0 (socket says 172.21.15.123:44488) 2023-11-15T06:37:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.500+0000 7fafaf7fe700 1 -- 172.21.15.123:0/2069276313 learned_addr learned my addr 172.21.15.123:0/2069276313 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafaf7fe700 1 -- 172.21.15.123:0/2069276313 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafb00fee90 con 0x7fafb0070370 2023-11-15T06:37:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafaf7fe700 1 --2- 172.21.15.123:0/2069276313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb00fe9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fafa400b5f0 tx=0x7fafa4006ee0).ready entity=mon.0 client_cookie=e86199509dae4345 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:08.506 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafae7fc700 1 -- 172.21.15.123:0/2069276313 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafa400d600 con 0x7fafb0070370 2023-11-15T06:37:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafae7fc700 1 -- 172.21.15.123:0/2069276313 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafa400dbd0 con 0x7fafb0070370 2023-11-15T06:37:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafae7fc700 1 -- 172.21.15.123:0/2069276313 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafa4011c00 con 0x7fafb0070370 2023-11-15T06:37:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafb5daf700 1 -- 172.21.15.123:0/2069276313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 msgr2=0x7fafb00fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafb5daf700 1 --2- 172.21.15.123:0/2069276313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb00fe9a0 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7fafa400b5f0 tx=0x7fafa4006ee0).stop 2023-11-15T06:37:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafb5daf700 1 -- 172.21.15.123:0/2069276313 shutdown_connections 2023-11-15T06:37:08.507 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafb5daf700 1 --2- 172.21.15.123:0/2069276313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb00fe9a0 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:08.508 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.501+0000 7fafb5daf700 1 -- 172.21.15.123:0/2069276313 >> 172.21.15.123:0/2069276313 conn(0x7fafb00f63c0 msgr2=0x7fafb00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:08.508 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.502+0000 7fafb5daf700 1 -- 172.21.15.123:0/2069276313 shutdown_connections 2023-11-15T06:37:08.510 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.502+0000 7fafb5daf700 1 -- 172.21.15.123:0/2069276313 wait complete. 2023-11-15T06:37:08.510 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.502+0000 7fafb5daf700 1 Processor -- start 2023-11-15T06:37:08.511 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.502+0000 7fafb5daf700 1 -- start start 2023-11-15T06:37:08.511 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.502+0000 7fafb5daf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb010aa10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:08.511 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.502+0000 7fafb5daf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafb010af00 con 0x7fafb0070370 2023-11-15T06:37:08.511 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7fafaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb010aa10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:08.511 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7fafaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb010aa10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44500/0 (socket says 172.21.15.123:44500) 2023-11-15T06:37:08.512 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7fafaf7fe700 1 -- 172.21.15.123:0/1468244834 learned_addr learned my addr 172.21.15.123:0/1468244834 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:08.512 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7fafaf7fe700 1 -- 172.21.15.123:0/1468244834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafa400b200 con 0x7fafb0070370 2023-11-15T06:37:08.512 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7fafaf7fe700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb010aa10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fafa400b1d0 tx=0x7fafa4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:08.512 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7faf97fff700 1 -- 172.21.15.123:0/1468244834 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafa4011a20 con 0x7fafb0070370 2023-11-15T06:37:08.512 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7faf97fff700 1 -- 172.21.15.123:0/1468244834 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafa401b880 con 0x7fafb0070370 2023-11-15T06:37:08.513 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafb010b100 con 0x7fafb0070370 2023-11-15T06:37:08.513 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7faf97fff700 1 -- 172.21.15.123:0/1468244834 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafa401a770 con 0x7fafb0070370 2023-11-15T06:37:08.513 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.503+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafb010b510 con 0x7fafb0070370 2023-11-15T06:37:08.513 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.504+0000 7faf97fff700 1 -- 172.21.15.123:0/1468244834 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7fafa400f070 con 0x7fafb0070370 2023-11-15T06:37:08.513 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.504+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafb0045ad0 con 0x7fafb0070370 2023-11-15T06:37:08.514 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.504+0000 7faf97fff700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7faf9c02fd10 0x7faf9c0321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:08.514 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.504+0000 7faf97fff700 1 -- 172.21.15.123:0/1468244834 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fafa4040f30 con 0x7fafb0070370 2023-11-15T06:37:08.514 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.504+0000 7fafaeffd700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7faf9c02fd10 0x7faf9c0321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:08.514 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.505+0000 7fafaeffd700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7faf9c02fd10 0x7faf9c0321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fafa00096a0 tx=0x7fafa0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:08.515 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.508+0000 7faf97fff700 1 -- 172.21.15.123:0/1468244834 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafa406d2b0 con 0x7fafb0070370 2023-11-15T06:37:08.686 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.683+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7fafb010bdf0 con 0x7faf9c02fd10 2023-11-15T06:37:08.695 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.692+0000 7faf97fff700 1 -- 172.21.15.123:0/1468244834 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7fafb010bdf0 con 0x7faf9c02fd10 2023-11-15T06:37:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7faf9c02fd10 msgr2=0x7faf9c0321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7faf9c02fd10 0x7faf9c0321d0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fafa00096a0 tx=0x7fafa0006b40).stop 2023-11-15T06:37:08.698 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 msgr2=0x7fafb010aa10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:08.699 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb010aa10 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fafa400b1d0 tx=0x7fafa4006ee0).stop 2023-11-15T06:37:08.699 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 shutdown_connections 2023-11-15T06:37:08.699 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7faf9c02fd10 0x7faf9c0321d0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:08.699 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 --2- 172.21.15.123:0/1468244834 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafb0070370 0x7fafb010aa10 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:08.700 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 >> 172.21.15.123:0/1468244834 conn(0x7fafb00f63c0 msgr2=0x7fafb00f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:08.700 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.694+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 shutdown_connections 2023-11-15T06:37:08.700 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:08.695+0000 7fafb5daf700 1 -- 172.21.15.123:0/1468244834 wait complete. 2023-11-15T06:37:09.030 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2023-11-15T06:37:09.151 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:08 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:07.743950+0000 mgr.smithi123.razjtn (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T06:37:09.151 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:08 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:07.744750+0000 mon.smithi123 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi123.razjtn(active, since 1.06288s) 2023-11-15T06:37:09.151 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:08 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:07.751706+0000 mgr.smithi123.razjtn (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T06:37:09.152 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:08 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:08.689412+0000 mon.smithi123 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:09.152 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:08 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:08.695794+0000 mon.smithi123 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:09.425 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f7d12d700 1 Processor -- start 2023-11-15T06:37:09.425 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f7d12d700 1 -- start start 2023-11-15T06:37:09.425 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f7d12d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:09.426 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f7d12d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780fedd0 con 0x7f6f78070430 2023-11-15T06:37:09.426 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f76d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:09.426 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f76d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44502/0 (socket says 172.21.15.123:44502) 2023-11-15T06:37:09.426 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f76d9d700 1 -- 172.21.15.123:0/1957362362 learned_addr learned my addr 172.21.15.123:0/1957362362 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:09.427 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.419+0000 7f6f76d9d700 1 -- 172.21.15.123:0/1957362362 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f780fef10 con 0x7f6f78070430 2023-11-15T06:37:09.427 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.420+0000 7f6f76d9d700 1 --2- 172.21.15.123:0/1957362362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fe8e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6f6800b5f0 tx=0x7f6f68006ee0).ready entity=mon.0 client_cookie=57dbc2e1da415c0c server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:09.427 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.420+0000 7f6f75d9b700 1 -- 172.21.15.123:0/1957362362 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f6800d600 con 0x7f6f78070430 2023-11-15T06:37:09.427 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.420+0000 7f6f75d9b700 1 -- 172.21.15.123:0/1957362362 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6f6800dbd0 con 0x7f6f78070430 2023-11-15T06:37:09.427 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.420+0000 7f6f75d9b700 1 -- 172.21.15.123:0/1957362362 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f68011c00 con 0x7f6f78070430 2023-11-15T06:37:09.427 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.420+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1957362362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 msgr2=0x7f6f780fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:09.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.420+0000 7f6f7d12d700 1 --2- 172.21.15.123:0/1957362362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fe8e0 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f6f6800b5f0 tx=0x7f6f68006ee0).stop 2023-11-15T06:37:09.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1957362362 shutdown_connections 2023-11-15T06:37:09.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 --2- 172.21.15.123:0/1957362362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fe8e0 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:09.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1957362362 >> 172.21.15.123:0/1957362362 conn(0x7f6f780f63c0 msgr2=0x7f6f780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:09.428 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1957362362 shutdown_connections 2023-11-15T06:37:09.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1957362362 wait complete. 2023-11-15T06:37:09.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 Processor -- start 2023-11-15T06:37:09.429 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 -- start start 2023-11-15T06:37:09.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:09.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.421+0000 7f6f7d12d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f780fa7e0 con 0x7f6f78070430 2023-11-15T06:37:09.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f76d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:09.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f76d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44504/0 (socket says 172.21.15.123:44504) 2023-11-15T06:37:09.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f76d9d700 1 -- 172.21.15.123:0/1047816657 learned_addr learned my addr 172.21.15.123:0/1047816657 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:09.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f76d9d700 1 -- 172.21.15.123:0/1047816657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f6800b200 con 0x7f6f78070430 2023-11-15T06:37:09.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f76d9d700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fc360 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6f6800b190 tx=0x7f6f68006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:09.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f5f7fe700 1 -- 172.21.15.123:0/1047816657 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f6801b410 con 0x7f6f78070430 2023-11-15T06:37:09.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f5f7fe700 1 -- 172.21.15.123:0/1047816657 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6f6801b9e0 con 0x7f6f78070430 2023-11-15T06:37:09.432 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f5f7fe700 1 -- 172.21.15.123:0/1047816657 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f6801a880 con 0x7f6f78070430 2023-11-15T06:37:09.432 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f780fa980 con 0x7f6f78070430 2023-11-15T06:37:09.432 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.422+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f780fad90 con 0x7f6f78070430 2023-11-15T06:37:09.432 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.423+0000 7f6f5f7fe700 1 -- 172.21.15.123:0/1047816657 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f6f6800f070 con 0x7f6f78070430 2023-11-15T06:37:09.433 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.423+0000 7f6f5f7fe700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f6f6402fcf0 0x7f6f640321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:09.433 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.423+0000 7f6f5f7fe700 1 -- 172.21.15.123:0/1047816657 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6f6801bb50 con 0x7f6f78070430 2023-11-15T06:37:09.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.423+0000 7f6f7659c700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f6f6402fcf0 0x7f6f640321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:09.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.423+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f78059100 con 0x7f6f78070430 2023-11-15T06:37:09.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.424+0000 7f6f7659c700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f6f6402fcf0 0x7f6f640321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6f6c0096a0 tx=0x7f6f6c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:09.434 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.427+0000 7f6f5f7fe700 1 -- 172.21.15.123:0/1047816657 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f68025460 con 0x7f6f78070430 2023-11-15T06:37:09.579 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.576+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f6f78104860 con 0x7f6f6402fcf0 2023-11-15T06:37:09.580 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.577+0000 7f6f5f7fe700 1 -- 172.21.15.123:0/1047816657 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f6f78104860 con 0x7f6f6402fcf0 2023-11-15T06:37:09.580 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: value unchanged 2023-11-15T06:37:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f6f6402fcf0 msgr2=0x7f6f640321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:09.582 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f6f6402fcf0 0x7f6f640321b0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f6f6c0096a0 tx=0x7f6f6c006b40).stop 2023-11-15T06:37:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 msgr2=0x7f6f780fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fc360 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f6f6800b190 tx=0x7f6f68006ee0).stop 2023-11-15T06:37:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 shutdown_connections 2023-11-15T06:37:09.583 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f6f6402fcf0 0x7f6f640321b0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 --2- 172.21.15.123:0/1047816657 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f78070430 0x7f6f780fc360 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 >> 172.21.15.123:0/1047816657 conn(0x7f6f780f63c0 msgr2=0x7f6f780f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 shutdown_connections 2023-11-15T06:37:09.584 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:09.579+0000 7f6f7d12d700 1 -- 172.21.15.123:0/1047816657 wait complete. 2023-11-15T06:37:09.882 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:08.685411+0000 mgr.smithi123.razjtn (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:09.882 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:09.690378+0000 mon.smithi123 (mon.0) 51 : cluster [DBG] mgrmap e8: smithi123.razjtn(active, since 3s) 2023-11-15T06:37:09.938 INFO:teuthology.orchestra.run.smithi123.stderr:Generating ssh key... 2023-11-15T06:37:09.939 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2023-11-15T06:37:10.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.285+0000 7f0f21b6b700 1 Processor -- start 2023-11-15T06:37:10.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.285+0000 7f0f21b6b700 1 -- start start 2023-11-15T06:37:10.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.285+0000 7f0f21b6b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c0fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:10.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.285+0000 7f0f21b6b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c0fedd0 con 0x7f0f1c070430 2023-11-15T06:37:10.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c0fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:10.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c0fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44514/0 (socket says 172.21.15.123:44514) 2023-11-15T06:37:10.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1b7fe700 1 -- 172.21.15.123:0/2182168721 learned_addr learned my addr 172.21.15.123:0/2182168721 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:10.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1b7fe700 1 -- 172.21.15.123:0/2182168721 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1c0fef10 con 0x7f0f1c070430 2023-11-15T06:37:10.293 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1b7fe700 1 --2- 172.21.15.123:0/2182168721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c0fe8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0f1000b5f0 tx=0x7f0f10006ee0).ready entity=mon.0 client_cookie=ff5ddebdda217500 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:10.294 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1a7fc700 1 -- 172.21.15.123:0/2182168721 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1000d600 con 0x7f0f1c070430 2023-11-15T06:37:10.294 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1a7fc700 1 -- 172.21.15.123:0/2182168721 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0f1000dbd0 con 0x7f0f1c070430 2023-11-15T06:37:10.294 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.286+0000 7f0f1a7fc700 1 -- 172.21.15.123:0/2182168721 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f10011c00 con 0x7f0f1c070430 2023-11-15T06:37:10.294 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2182168721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 msgr2=0x7f0f1c0fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:10.294 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 --2- 172.21.15.123:0/2182168721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c0fe8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f0f1000b5f0 tx=0x7f0f10006ee0).stop 2023-11-15T06:37:10.295 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2182168721 shutdown_connections 2023-11-15T06:37:10.295 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 --2- 172.21.15.123:0/2182168721 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c0fe8e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:10.295 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2182168721 >> 172.21.15.123:0/2182168721 conn(0x7f0f1c0f63c0 msgr2=0x7f0f1c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:10.295 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2182168721 shutdown_connections 2023-11-15T06:37:10.295 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2182168721 wait complete. 2023-11-15T06:37:10.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.287+0000 7f0f21b6b700 1 Processor -- start 2023-11-15T06:37:10.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f21b6b700 1 -- start start 2023-11-15T06:37:10.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f21b6b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:10.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f21b6b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f1c10af30 con 0x7f0f1c070430 2023-11-15T06:37:10.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f1b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:10.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f1b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44528/0 (socket says 172.21.15.123:44528) 2023-11-15T06:37:10.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f1b7fe700 1 -- 172.21.15.123:0/2584432203 learned_addr learned my addr 172.21.15.123:0/2584432203 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:10.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f1b7fe700 1 -- 172.21.15.123:0/2584432203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f1000b200 con 0x7f0f1c070430 2023-11-15T06:37:10.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.288+0000 7f0f1b7fe700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c10aa40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0f1000e040 tx=0x7f0f10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:10.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.289+0000 7f0f03fff700 1 -- 172.21.15.123:0/2584432203 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1000d9f0 con 0x7f0f1c070430 2023-11-15T06:37:10.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.289+0000 7f0f03fff700 1 -- 172.21.15.123:0/2584432203 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f0f1001a9f0 con 0x7f0f1c070430 2023-11-15T06:37:10.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.289+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f1c10b130 con 0x7f0f1c070430 2023-11-15T06:37:10.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.289+0000 7f0f03fff700 1 -- 172.21.15.123:0/2584432203 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f1001b8c0 con 0x7f0f1c070430 2023-11-15T06:37:10.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.289+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f1c10b540 con 0x7f0f1c070430 2023-11-15T06:37:10.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.289+0000 7f0f03fff700 1 -- 172.21.15.123:0/2584432203 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f0f1000f070 con 0x7f0f1c070430 2023-11-15T06:37:10.299 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.289+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f1c045ad0 con 0x7f0f1c070430 2023-11-15T06:37:10.299 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.290+0000 7f0f03fff700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0f0802fd50 0x7f0f08032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:10.299 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.290+0000 7f0f03fff700 1 -- 172.21.15.123:0/2584432203 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0f100412f0 con 0x7f0f1c070430 2023-11-15T06:37:10.299 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.290+0000 7f0f1affd700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0f0802fd50 0x7f0f08032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:10.299 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.290+0000 7f0f1affd700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0f0802fd50 0x7f0f08032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0f0c0096a0 tx=0x7f0f0c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:10.300 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.294+0000 7f0f03fff700 1 -- 172.21.15.123:0/2584432203 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f10022d70 con 0x7f0f1c070430 2023-11-15T06:37:10.444 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.442+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f0f1c10be80 con 0x7f0f0802fd50 2023-11-15T06:37:10.626 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.624+0000 7f0f03fff700 1 -- 172.21.15.123:0/2584432203 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f0f1c10be80 con 0x7f0f0802fd50 2023-11-15T06:37:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0f0802fd50 msgr2=0x7f0f08032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0f0802fd50 0x7f0f08032210 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f0f0c0096a0 tx=0x7f0f0c006b40).stop 2023-11-15T06:37:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 msgr2=0x7f0f1c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:10.629 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c10aa40 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f0f1000e040 tx=0x7f0f10006ee0).stop 2023-11-15T06:37:10.630 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 shutdown_connections 2023-11-15T06:37:10.630 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0f0802fd50 0x7f0f08032210 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:10.630 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 --2- 172.21.15.123:0/2584432203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0f1c070430 0x7f0f1c10aa40 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:10.630 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 >> 172.21.15.123:0/2584432203 conn(0x7f0f1c0f63c0 msgr2=0x7f0f1c0f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:10.630 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 shutdown_connections 2023-11-15T06:37:10.631 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:10.626+0000 7f0f21b6b700 1 -- 172.21.15.123:0/2584432203 wait complete. 2023-11-15T06:37:10.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:10 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:09.578103+0000 mgr.smithi123.razjtn (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:10.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:10 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:10.621964+0000 mon.smithi123 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:10.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:10 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:10.624266+0000 mon.smithi123 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:10.971 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2023-11-15T06:37:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.301+0000 7fafce875700 1 Processor -- start 2023-11-15T06:37:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.301+0000 7fafce875700 1 -- start start 2023-11-15T06:37:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.301+0000 7fafce875700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc80fe9a0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:11.307 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.301+0000 7fafce875700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc8070790 con 0x7fafc8070370 2023-11-15T06:37:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.301+0000 7fafc7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc80fe9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.301+0000 7fafc7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc80fe9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44530/0 (socket says 172.21.15.123:44530) 2023-11-15T06:37:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.301+0000 7fafc7fff700 1 -- 172.21.15.123:0/2628621906 learned_addr learned my addr 172.21.15.123:0/2628621906 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafc7fff700 1 -- 172.21.15.123:0/2628621906 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafc80fee90 con 0x7fafc8070370 2023-11-15T06:37:11.308 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafc7fff700 1 --2- 172.21.15.123:0/2628621906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc80fe9a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fafbc00b5f0 tx=0x7fafbc006ee0).ready entity=mon.0 client_cookie=ce0ee3d0aee4ef9f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafc6ffd700 1 -- 172.21.15.123:0/2628621906 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafbc00d600 con 0x7fafc8070370 2023-11-15T06:37:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafc6ffd700 1 -- 172.21.15.123:0/2628621906 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafbc00dbd0 con 0x7fafc8070370 2023-11-15T06:37:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafc6ffd700 1 -- 172.21.15.123:0/2628621906 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafbc011c00 con 0x7fafc8070370 2023-11-15T06:37:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafce875700 1 -- 172.21.15.123:0/2628621906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 msgr2=0x7fafc80fe9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:11.309 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafce875700 1 --2- 172.21.15.123:0/2628621906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc80fe9a0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7fafbc00b5f0 tx=0x7fafbc006ee0).stop 2023-11-15T06:37:11.310 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafce875700 1 -- 172.21.15.123:0/2628621906 shutdown_connections 2023-11-15T06:37:11.310 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafce875700 1 --2- 172.21.15.123:0/2628621906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc80fe9a0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:11.310 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.302+0000 7fafce875700 1 -- 172.21.15.123:0/2628621906 >> 172.21.15.123:0/2628621906 conn(0x7fafc80f63c0 msgr2=0x7fafc80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:11.310 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.303+0000 7fafce875700 1 -- 172.21.15.123:0/2628621906 shutdown_connections 2023-11-15T06:37:11.310 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.303+0000 7fafce875700 1 -- 172.21.15.123:0/2628621906 wait complete. 2023-11-15T06:37:11.311 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.303+0000 7fafce875700 1 Processor -- start 2023-11-15T06:37:11.311 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.303+0000 7fafce875700 1 -- start start 2023-11-15T06:37:11.311 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.303+0000 7fafce875700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc810a9b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:11.311 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.303+0000 7fafce875700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fafc810aea0 con 0x7fafc8070370 2023-11-15T06:37:11.311 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc810a9b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:11.312 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc810a9b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44544/0 (socket says 172.21.15.123:44544) 2023-11-15T06:37:11.312 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc7fff700 1 -- 172.21.15.123:0/1933580971 learned_addr learned my addr 172.21.15.123:0/1933580971 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:11.312 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc7fff700 1 -- 172.21.15.123:0/1933580971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fafbc00b200 con 0x7fafc8070370 2023-11-15T06:37:11.312 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc7fff700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc810a9b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fafbc00b190 tx=0x7fafbc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:11.312 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc4ff9700 1 -- 172.21.15.123:0/1933580971 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafbc01b410 con 0x7fafc8070370 2023-11-15T06:37:11.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc4ff9700 1 -- 172.21.15.123:0/1933580971 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fafbc01b9e0 con 0x7fafc8070370 2023-11-15T06:37:11.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fafc810b0a0 con 0x7fafc8070370 2023-11-15T06:37:11.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafc4ff9700 1 -- 172.21.15.123:0/1933580971 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fafbc01a8f0 con 0x7fafc8070370 2023-11-15T06:37:11.313 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.304+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fafc810b4b0 con 0x7fafc8070370 2023-11-15T06:37:11.314 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.305+0000 7fafc4ff9700 1 -- 172.21.15.123:0/1933580971 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fafbc00f070 con 0x7fafc8070370 2023-11-15T06:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.305+0000 7fafc4ff9700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fafb402fdb0 0x7fafb4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.305+0000 7fafc4ff9700 1 -- 172.21.15.123:0/1933580971 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fafbc041220 con 0x7fafc8070370 2023-11-15T06:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.305+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fafc8045ad0 con 0x7fafc8070370 2023-11-15T06:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.305+0000 7fafc77fe700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fafb402fdb0 0x7fafb4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:11.315 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.306+0000 7fafc77fe700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fafb402fdb0 0x7fafb4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fafb80096a0 tx=0x7fafb8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:11.316 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.309+0000 7fafc4ff9700 1 -- 172.21.15.123:0/1933580971 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fafbc016070 con 0x7fafc8070370 2023-11-15T06:37:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.461+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7fafc81048a0 con 0x7fafb402fdb0 2023-11-15T06:37:11.464 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.462+0000 7fafc4ff9700 1 -- 172.21.15.123:0/1933580971 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7fafc81048a0 con 0x7fafb402fdb0 2023-11-15T06:37:11.465 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbbBRJvpr3wJrixHcdvgQOVMeGSYrjFeIM8VMEQTQ+WUkT9kF5yep0whX/uD/sr6Zae8XhvejMRhbvovoHqHMCUx/TADUSeFm4R9ntKnrtAjybfy519BKJV7IjYA+a1mPA7Q9x+TZlxbVRFmixugoHMZF5yoWXiFX7qWTxsPvgUTF7I1Fl37B+zGVodA9IAZqOwtSeEqUKwLhI3FvI18ZDOQ9IEKkmaHjS2dRqQd3flORGF2dX/dr//ezJifMRADAMkpyZDGE2NdlXrrM358or09gtRtPvHqqczy4qrSzV128tDCxvGvgab7TEdYFAgUKL+YaAW/iKNeaG2pixhhsVtCXS+2Tg0h9rGZ2uIfs/Ab3jTj3e3fLRZ8sqtQ0nSfZRo8ySWdc61aDvhGUKTd+tdPg/N7AJLgZDgZGIzgIhQdT9mp7fclKup/fs11zURCQSeTROpKO3drZ/c5bDG+TBDt4YvObR5gSfiF6XtRbkv+rBf6o2BXwNy+hE29RXlU0= ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:37:11.467 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fafb402fdb0 msgr2=0x7fafb4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:11.467 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fafb402fdb0 0x7fafb4032270 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fafb80096a0 tx=0x7fafb8006b40).stop 2023-11-15T06:37:11.467 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 msgr2=0x7fafc810a9b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:11.467 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc810a9b0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fafbc00b190 tx=0x7fafbc006ee0).stop 2023-11-15T06:37:11.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 shutdown_connections 2023-11-15T06:37:11.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fafb402fdb0 0x7fafb4032270 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:11.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 --2- 172.21.15.123:0/1933580971 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fafc8070370 0x7fafc810a9b0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:11.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 >> 172.21.15.123:0/1933580971 conn(0x7fafc80f63c0 msgr2=0x7fafc80f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:11.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 shutdown_connections 2023-11-15T06:37:11.469 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:11.464+0000 7fafce875700 1 -- 172.21.15.123:0/1933580971 wait complete. 2023-11-15T06:37:11.821 INFO:teuthology.orchestra.run.smithi123.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:37:11.821 INFO:teuthology.orchestra.run.smithi123.stderr:Adding key to root@localhost authorized_keys... 2023-11-15T06:37:11.822 INFO:teuthology.orchestra.run.smithi123.stderr:Adding host smithi123... 2023-11-15T06:37:11.822 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi123 172.21.15.123 2023-11-15T06:37:12.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:10.443434+0000 mgr.smithi123.razjtn (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2023-11-15T06:37:13.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:12 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:11.462933+0000 mgr.smithi123.razjtn (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:13.412 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.406+0000 7f8e54dbe700 1 Processor -- start 2023-11-15T06:37:13.412 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.406+0000 7f8e54dbe700 1 -- start start 2023-11-15T06:37:13.413 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.407+0000 7f8e54dbe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:13.413 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.407+0000 7f8e54dbe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500fee30 con 0x7f8e500703d0 2023-11-15T06:37:13.413 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.407+0000 7f8e4e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:13.413 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.407+0000 7f8e4e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44558/0 (socket says 172.21.15.123:44558) 2023-11-15T06:37:13.413 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.407+0000 7f8e4e59c700 1 -- 172.21.15.123:0/3693970551 learned_addr learned my addr 172.21.15.123:0/3693970551 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:13.414 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.407+0000 7f8e4e59c700 1 -- 172.21.15.123:0/3693970551 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e500fef70 con 0x7f8e500703d0 2023-11-15T06:37:13.414 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.407+0000 7f8e4e59c700 1 --2- 172.21.15.123:0/3693970551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fe940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8e4400b5f0 tx=0x7f8e44006ee0).ready entity=mon.0 client_cookie=c6a5b2d13669ffaa server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:13.414 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.408+0000 7f8e4d59a700 1 -- 172.21.15.123:0/3693970551 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e4400d600 con 0x7f8e500703d0 2023-11-15T06:37:13.414 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.408+0000 7f8e4d59a700 1 -- 172.21.15.123:0/3693970551 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e4400dbd0 con 0x7f8e500703d0 2023-11-15T06:37:13.415 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.408+0000 7f8e4d59a700 1 -- 172.21.15.123:0/3693970551 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e44011c00 con 0x7f8e500703d0 2023-11-15T06:37:13.415 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.408+0000 7f8e54dbe700 1 -- 172.21.15.123:0/3693970551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 msgr2=0x7f8e500fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:13.415 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.408+0000 7f8e54dbe700 1 --2- 172.21.15.123:0/3693970551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fe940 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8e4400b5f0 tx=0x7f8e44006ee0).stop 2023-11-15T06:37:13.415 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.408+0000 7f8e54dbe700 1 -- 172.21.15.123:0/3693970551 shutdown_connections 2023-11-15T06:37:13.415 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.408+0000 7f8e54dbe700 1 --2- 172.21.15.123:0/3693970551 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fe940 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:13.416 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.409+0000 7f8e54dbe700 1 -- 172.21.15.123:0/3693970551 >> 172.21.15.123:0/3693970551 conn(0x7f8e500f63c0 msgr2=0x7f8e500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:13.416 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.409+0000 7f8e54dbe700 1 -- 172.21.15.123:0/3693970551 shutdown_connections 2023-11-15T06:37:13.416 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.409+0000 7f8e54dbe700 1 -- 172.21.15.123:0/3693970551 wait complete. 2023-11-15T06:37:13.416 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.409+0000 7f8e54dbe700 1 Processor -- start 2023-11-15T06:37:13.417 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.409+0000 7f8e54dbe700 1 -- start start 2023-11-15T06:37:13.417 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e54dbe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fc3c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:13.417 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e54dbe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8e500fa840 con 0x7f8e500703d0 2023-11-15T06:37:13.418 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e4e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fc3c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:13.419 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e4e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fc3c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:44564/0 (socket says 172.21.15.123:44564) 2023-11-15T06:37:13.419 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e4e59c700 1 -- 172.21.15.123:0/119530826 learned_addr learned my addr 172.21.15.123:0/119530826 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:13.419 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e4e59c700 1 -- 172.21.15.123:0/119530826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e4400b200 con 0x7f8e500703d0 2023-11-15T06:37:13.419 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e4e59c700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fc3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8e4400e040 tx=0x7f8e44006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:13.419 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e3effd700 1 -- 172.21.15.123:0/119530826 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e4400d9f0 con 0x7f8e500703d0 2023-11-15T06:37:13.420 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e3effd700 1 -- 172.21.15.123:0/119530826 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f8e4401a9f0 con 0x7f8e500703d0 2023-11-15T06:37:13.420 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8e500fa9e0 con 0x7f8e500703d0 2023-11-15T06:37:13.420 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e3effd700 1 -- 172.21.15.123:0/119530826 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e4401b8c0 con 0x7f8e500703d0 2023-11-15T06:37:13.420 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.410+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8e500fadf0 con 0x7f8e500703d0 2023-11-15T06:37:13.420 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.411+0000 7f8e3effd700 1 -- 172.21.15.123:0/119530826 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f8e4400f070 con 0x7f8e500703d0 2023-11-15T06:37:13.421 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.411+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8e50045ad0 con 0x7f8e500703d0 2023-11-15T06:37:13.421 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.411+0000 7f8e3effd700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f8e3402fdb0 0x7f8e34032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:13.421 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.412+0000 7f8e3effd700 1 -- 172.21.15.123:0/119530826 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f8e440412c0 con 0x7f8e500703d0 2023-11-15T06:37:13.421 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.412+0000 7f8e4dd9b700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f8e3402fdb0 0x7f8e34032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:13.421 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.412+0000 7f8e4dd9b700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f8e3402fdb0 0x7f8e34032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8e400096a0 tx=0x7f8e40006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:13.422 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.416+0000 7f8e3effd700 1 -- 172.21.15.123:0/119530826 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e44019d20 con 0x7f8e500703d0 2023-11-15T06:37:13.594 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:13.591+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi123", "addr": "172.21.15.123", "target": ["mon-mgr", ""]}) v1 -- 0x7f8e500fb030 con 0x7f8e3402fdb0 2023-11-15T06:37:13.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:13.593077+0000 mgr.smithi123.razjtn (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi123", "addr": "172.21.15.123", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:14.218 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Added host 'smithi123' 2023-11-15T06:37:14.218 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.215+0000 7f8e3effd700 1 -- 172.21.15.123:0/119530826 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f8e500fb030 con 0x7f8e3402fdb0 2023-11-15T06:37:14.220 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.217+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f8e3402fdb0 msgr2=0x7f8e34032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:14.220 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.217+0000 7f8e54dbe700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f8e3402fdb0 0x7f8e34032270 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f8e400096a0 tx=0x7f8e40006b40).stop 2023-11-15T06:37:14.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.217+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 msgr2=0x7f8e500fc3c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:14.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.217+0000 7f8e54dbe700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fc3c0 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f8e4400e040 tx=0x7f8e44006ee0).stop 2023-11-15T06:37:14.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.218+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 shutdown_connections 2023-11-15T06:37:14.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.218+0000 7f8e54dbe700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f8e3402fdb0 0x7f8e34032270 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:14.222 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.218+0000 7f8e54dbe700 1 --2- 172.21.15.123:0/119530826 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8e500703d0 0x7f8e500fc3c0 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:14.222 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.218+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 >> 172.21.15.123:0/119530826 conn(0x7f8e500f63c0 msgr2=0x7f8e500f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:14.222 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.218+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 shutdown_connections 2023-11-15T06:37:14.222 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:14.218+0000 7f8e54dbe700 1 -- 172.21.15.123:0/119530826 wait complete. 2023-11-15T06:37:15.657 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:14.215139+0000 mon.smithi123 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:15.658 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:14.215445+0000 mgr.smithi123.razjtn (mgr.14114) 9 : cephadm [INF] Added host smithi123 2023-11-15T06:37:15.658 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:14.215666+0000 mon.smithi123 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:16.377 INFO:teuthology.orchestra.run.smithi123.stderr:Deploying mon service with default placement... 2023-11-15T06:37:16.378 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2023-11-15T06:37:17.785 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.708+0000 7fc3ff223700 1 Processor -- start 2023-11-15T06:37:17.785 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.708+0000 7fc3ff223700 1 -- start start 2023-11-15T06:37:17.786 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.708+0000 7fc3ff223700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f80fc550 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:17.786 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.708+0000 7fc3ff223700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f80fca40 con 0x7fc3f80fc130 2023-11-15T06:37:17.787 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.709+0000 7fc3fe221700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f80fc550 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:17.787 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.709+0000 7fc3fe221700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f80fc550 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43060/0 (socket says 172.21.15.123:43060) 2023-11-15T06:37:17.787 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.709+0000 7fc3fe221700 1 -- 172.21.15.123:0/1765269800 learned_addr learned my addr 172.21.15.123:0/1765269800 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:17.787 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.709+0000 7fc3fe221700 1 -- 172.21.15.123:0/1765269800 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3f80fcb80 con 0x7fc3f80fc130 2023-11-15T06:37:17.788 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.709+0000 7fc3fe221700 1 --2- 172.21.15.123:0/1765269800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f80fc550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fc3e800b5f0 tx=0x7fc3e8006ee0).ready entity=mon.0 client_cookie=1639674c4c745a10 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:17.788 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.709+0000 7fc3fd21f700 1 -- 172.21.15.123:0/1765269800 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e800d600 con 0x7fc3f80fc130 2023-11-15T06:37:17.788 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3fd21f700 1 -- 172.21.15.123:0/1765269800 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc3e800dbd0 con 0x7fc3f80fc130 2023-11-15T06:37:17.788 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3fd21f700 1 -- 172.21.15.123:0/1765269800 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e8011c00 con 0x7fc3f80fc130 2023-11-15T06:37:17.789 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3ff223700 1 -- 172.21.15.123:0/1765269800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 msgr2=0x7fc3f80fc550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:17.789 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3ff223700 1 --2- 172.21.15.123:0/1765269800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f80fc550 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fc3e800b5f0 tx=0x7fc3e8006ee0).stop 2023-11-15T06:37:17.789 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3ff223700 1 -- 172.21.15.123:0/1765269800 shutdown_connections 2023-11-15T06:37:17.789 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3ff223700 1 --2- 172.21.15.123:0/1765269800 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f80fc550 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:17.789 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3ff223700 1 -- 172.21.15.123:0/1765269800 >> 172.21.15.123:0/1765269800 conn(0x7fc3f80f59a0 msgr2=0x7fc3f80f7dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:17.790 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3ff223700 1 -- 172.21.15.123:0/1765269800 shutdown_connections 2023-11-15T06:37:17.790 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.710+0000 7fc3ff223700 1 -- 172.21.15.123:0/1765269800 wait complete. 2023-11-15T06:37:17.791 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3ff223700 1 Processor -- start 2023-11-15T06:37:17.792 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3ff223700 1 -- start start 2023-11-15T06:37:17.792 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3ff223700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f810ca80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:17.792 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3ff223700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3f810cf70 con 0x7fc3f80fc130 2023-11-15T06:37:17.792 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3fe221700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f810ca80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:17.792 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3fe221700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f810ca80 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43068/0 (socket says 172.21.15.123:43068) 2023-11-15T06:37:17.793 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3fe221700 1 -- 172.21.15.123:0/1282902560 learned_addr learned my addr 172.21.15.123:0/1282902560 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:17.793 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3fe221700 1 -- 172.21.15.123:0/1282902560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3e800b200 con 0x7fc3f80fc130 2023-11-15T06:37:17.793 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3fe221700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f810ca80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc3e8000c00 tx=0x7fc3e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:17.793 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.711+0000 7fc3e6ffd700 1 -- 172.21.15.123:0/1282902560 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e8011690 con 0x7fc3f80fc130 2023-11-15T06:37:17.793 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.712+0000 7fc3e6ffd700 1 -- 172.21.15.123:0/1282902560 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc3e800d9f0 con 0x7fc3f80fc130 2023-11-15T06:37:17.794 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.712+0000 7fc3e6ffd700 1 -- 172.21.15.123:0/1282902560 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3e801b710 con 0x7fc3f80fc130 2023-11-15T06:37:17.794 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.712+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3f810d170 con 0x7fc3f80fc130 2023-11-15T06:37:17.794 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.712+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3f810d580 con 0x7fc3f80fc130 2023-11-15T06:37:17.794 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.712+0000 7fc3e6ffd700 1 -- 172.21.15.123:0/1282902560 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fc3e800f070 con 0x7fc3f80fc130 2023-11-15T06:37:17.794 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.712+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3f8045ad0 con 0x7fc3f80fc130 2023-11-15T06:37:17.795 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.713+0000 7fc3e6ffd700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fc3e002fdb0 0x7fc3e0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:17.795 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.713+0000 7fc3e6ffd700 1 -- 172.21.15.123:0/1282902560 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc3e8041200 con 0x7fc3f80fc130 2023-11-15T06:37:17.795 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.713+0000 7fc3fda20700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fc3e002fdb0 0x7fc3e0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:17.795 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.713+0000 7fc3fda20700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fc3e002fdb0 0x7fc3e0032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc3f40096a0 tx=0x7fc3f4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:17.796 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.717+0000 7fc3e6ffd700 1 -- 172.21.15.123:0/1282902560 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3e8022900 con 0x7fc3f80fc130 2023-11-15T06:37:17.898 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.895+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7fc3f810de60 con 0x7fc3e002fdb0 2023-11-15T06:37:17.902 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.900+0000 7fc3e6ffd700 1 -- 172.21.15.123:0/1282902560 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fc3f810de60 con 0x7fc3e002fdb0 2023-11-15T06:37:17.902 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Scheduled mon update... 2023-11-15T06:37:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fc3e002fdb0 msgr2=0x7fc3e0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fc3e002fdb0 0x7fc3e0032270 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7fc3f40096a0 tx=0x7fc3f4006b40).stop 2023-11-15T06:37:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 msgr2=0x7fc3f810ca80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f810ca80 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fc3e8000c00 tx=0x7fc3e8006ee0).stop 2023-11-15T06:37:17.905 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 shutdown_connections 2023-11-15T06:37:17.906 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fc3e002fdb0 0x7fc3e0032270 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:17.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 --2- 172.21.15.123:0/1282902560 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc3f80fc130 0x7fc3f810ca80 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:17.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 >> 172.21.15.123:0/1282902560 conn(0x7fc3f80f59a0 msgr2=0x7fc3f80f74f0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:17.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 shutdown_connections 2023-11-15T06:37:17.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:17.902+0000 7fc3ff223700 1 -- 172.21.15.123:0/1282902560 wait complete. 2023-11-15T06:37:19.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:18 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:17.897076+0000 mgr.smithi123.razjtn (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:19.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:18 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:17.897789+0000 mgr.smithi123.razjtn (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2023-11-15T06:37:19.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:18 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:17.900707+0000 mon.smithi123 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:19.520 INFO:teuthology.orchestra.run.smithi123.stderr:Deploying mgr service with default placement... 2023-11-15T06:37:19.521 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2023-11-15T06:37:19.907 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.901+0000 7f27da5d5700 1 Processor -- start 2023-11-15T06:37:19.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.901+0000 7f27da5d5700 1 -- start start 2023-11-15T06:37:19.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.901+0000 7f27da5d5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d4100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:19.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.901+0000 7f27da5d5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d40fc750 con 0x7f27d40703d0 2023-11-15T06:37:19.908 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d4100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:19.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d4100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43084/0 (socket says 172.21.15.123:43084) 2023-11-15T06:37:19.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d3fff700 1 -- 172.21.15.123:0/515417817 learned_addr learned my addr 172.21.15.123:0/515417817 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:19.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d3fff700 1 -- 172.21.15.123:0/515417817 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27d4101250 con 0x7f27d40703d0 2023-11-15T06:37:19.909 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d3fff700 1 --2- 172.21.15.123:0/515417817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d4100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f27c800b5f0 tx=0x7f27c8006ee0).ready entity=mon.0 client_cookie=1bfd4fdd3717cce4 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:19.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d2ffd700 1 -- 172.21.15.123:0/515417817 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c800d600 con 0x7f27d40703d0 2023-11-15T06:37:19.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d2ffd700 1 -- 172.21.15.123:0/515417817 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f27c800dbd0 con 0x7f27d40703d0 2023-11-15T06:37:19.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.902+0000 7f27d2ffd700 1 -- 172.21.15.123:0/515417817 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c8011c00 con 0x7f27d40703d0 2023-11-15T06:37:19.910 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.903+0000 7f27da5d5700 1 -- 172.21.15.123:0/515417817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 msgr2=0x7f27d4100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:19.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.903+0000 7f27da5d5700 1 --2- 172.21.15.123:0/515417817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d4100d60 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f27c800b5f0 tx=0x7f27c8006ee0).stop 2023-11-15T06:37:19.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.903+0000 7f27da5d5700 1 -- 172.21.15.123:0/515417817 shutdown_connections 2023-11-15T06:37:19.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.903+0000 7f27da5d5700 1 --2- 172.21.15.123:0/515417817 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d4100d60 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:19.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.903+0000 7f27da5d5700 1 -- 172.21.15.123:0/515417817 >> 172.21.15.123:0/515417817 conn(0x7f27d40f63c0 msgr2=0x7f27d40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:19.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.903+0000 7f27da5d5700 1 -- 172.21.15.123:0/515417817 shutdown_connections 2023-11-15T06:37:19.911 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.903+0000 7f27da5d5700 1 -- 172.21.15.123:0/515417817 wait complete. 2023-11-15T06:37:19.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27da5d5700 1 Processor -- start 2023-11-15T06:37:19.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27da5d5700 1 -- start start 2023-11-15T06:37:19.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27da5d5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d410cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:19.912 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27da5d5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f27d410d0f0 con 0x7f27d40703d0 2023-11-15T06:37:19.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27d3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d410cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:19.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27d3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d410cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43092/0 (socket says 172.21.15.123:43092) 2023-11-15T06:37:19.913 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27d3fff700 1 -- 172.21.15.123:0/885900808 learned_addr learned my addr 172.21.15.123:0/885900808 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:19.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27d3fff700 1 -- 172.21.15.123:0/885900808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f27c800b200 con 0x7f27d40703d0 2023-11-15T06:37:19.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.904+0000 7f27d3fff700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d410cc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f27c800b190 tx=0x7f27c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:19.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.905+0000 7f27d0ff9700 1 -- 172.21.15.123:0/885900808 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c801b410 con 0x7f27d40703d0 2023-11-15T06:37:19.915 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.905+0000 7f27d0ff9700 1 -- 172.21.15.123:0/885900808 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f27c801b9e0 con 0x7f27d40703d0 2023-11-15T06:37:19.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.905+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f27d410d2f0 con 0x7f27d40703d0 2023-11-15T06:37:19.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.905+0000 7f27d0ff9700 1 -- 172.21.15.123:0/885900808 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f27c801a880 con 0x7f27d40703d0 2023-11-15T06:37:19.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.905+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f27d410d700 con 0x7f27d40703d0 2023-11-15T06:37:19.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.905+0000 7f27d0ff9700 1 -- 172.21.15.123:0/885900808 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f27c800f070 con 0x7f27d40703d0 2023-11-15T06:37:19.916 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.906+0000 7f27d0ff9700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f27c002fd50 0x7f27c0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:19.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.906+0000 7f27d0ff9700 1 -- 172.21.15.123:0/885900808 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f27c80412b0 con 0x7f27d40703d0 2023-11-15T06:37:19.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.906+0000 7f27d37fe700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f27c002fd50 0x7f27c0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:19.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.906+0000 7f27d37fe700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f27c002fd50 0x7f27c0032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f27c40096a0 tx=0x7f27c4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:19.917 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.906+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f27d4059100 con 0x7f27d40703d0 2023-11-15T06:37:19.918 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:19.910+0000 7f27d0ff9700 1 -- 172.21.15.123:0/885900808 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f27c8010360 con 0x7f27d40703d0 2023-11-15T06:37:20.086 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.084+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7f27d4106ae0 con 0x7f27c002fd50 2023-11-15T06:37:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.087+0000 7f27d0ff9700 1 -- 172.21.15.123:0/885900808 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f27d4106ae0 con 0x7f27c002fd50 2023-11-15T06:37:20.090 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Scheduled mgr update... 2023-11-15T06:37:20.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.089+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f27c002fd50 msgr2=0x7f27c0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:20.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.089+0000 7f27da5d5700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f27c002fd50 0x7f27c0032210 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f27c40096a0 tx=0x7f27c4006b40).stop 2023-11-15T06:37:20.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.089+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 msgr2=0x7f27d410cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:20.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.089+0000 7f27da5d5700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d410cc00 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f27c800b190 tx=0x7f27c8006ee0).stop 2023-11-15T06:37:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.089+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 shutdown_connections 2023-11-15T06:37:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.090+0000 7f27da5d5700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f27c002fd50 0x7f27c0032210 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.090+0000 7f27da5d5700 1 --2- 172.21.15.123:0/885900808 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f27d40703d0 0x7f27d410cc00 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.090+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 >> 172.21.15.123:0/885900808 conn(0x7f27d40f63c0 msgr2=0x7f27d40f7e50 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:20.095 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.090+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 shutdown_connections 2023-11-15T06:37:20.096 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.090+0000 7f27da5d5700 1 -- 172.21.15.123:0/885900808 wait complete. 2023-11-15T06:37:20.445 INFO:teuthology.orchestra.run.smithi123.stderr:Deploying crash service with default placement... 2023-11-15T06:37:20.445 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2023-11-15T06:37:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f96b1700 1 Processor -- start 2023-11-15T06:37:20.851 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f96b1700 1 -- start start 2023-11-15T06:37:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f96b1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f96b1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f40fc5c0 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f2ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f2ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43100/0 (socket says 172.21.15.123:43100) 2023-11-15T06:37:20.852 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f2ffd700 1 -- 172.21.15.123:0/2688762386 learned_addr learned my addr 172.21.15.123:0/2688762386 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:20.853 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.847+0000 7fd7f2ffd700 1 -- 172.21.15.123:0/2688762386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7f40fc700 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.853 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f2ffd700 1 --2- 172.21.15.123:0/2688762386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40fc0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd7e800b5f0 tx=0x7fd7e8006ee0).ready entity=mon.0 client_cookie=2f66f3e2e2905c8a server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:20.853 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f1ffb700 1 -- 172.21.15.123:0/2688762386 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7e800d600 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.853 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f1ffb700 1 -- 172.21.15.123:0/2688762386 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd7e800dbd0 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.853 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f1ffb700 1 -- 172.21.15.123:0/2688762386 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7e8011c00 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f96b1700 1 -- 172.21.15.123:0/2688762386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 msgr2=0x7fd7f40fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f96b1700 1 --2- 172.21.15.123:0/2688762386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40fc0d0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7fd7e800b5f0 tx=0x7fd7e8006ee0).stop 2023-11-15T06:37:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f96b1700 1 -- 172.21.15.123:0/2688762386 shutdown_connections 2023-11-15T06:37:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f96b1700 1 --2- 172.21.15.123:0/2688762386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40fc0d0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:20.854 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.848+0000 7fd7f96b1700 1 -- 172.21.15.123:0/2688762386 >> 172.21.15.123:0/2688762386 conn(0x7fd7f40f74b0 msgr2=0x7fd7f40f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.849+0000 7fd7f96b1700 1 -- 172.21.15.123:0/2688762386 shutdown_connections 2023-11-15T06:37:20.855 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.849+0000 7fd7f96b1700 1 -- 172.21.15.123:0/2688762386 wait complete. 2023-11-15T06:37:20.857 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.849+0000 7fd7f96b1700 1 Processor -- start 2023-11-15T06:37:20.857 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.849+0000 7fd7f96b1700 1 -- start start 2023-11-15T06:37:20.857 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f96b1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:20.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f96b1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd7f4070a60 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f2ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:20.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f2ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43106/0 (socket says 172.21.15.123:43106) 2023-11-15T06:37:20.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f2ffd700 1 -- 172.21.15.123:0/1433998870 learned_addr learned my addr 172.21.15.123:0/1433998870 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:20.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f2ffd700 1 -- 172.21.15.123:0/1433998870 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd7e800b200 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f2ffd700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40725e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd7e800b1d0 tx=0x7fd7e8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:20.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7db7fe700 1 -- 172.21.15.123:0/1433998870 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7e801b410 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7db7fe700 1 -- 172.21.15.123:0/1433998870 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fd7e801b9e0 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd7f4070c00 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7db7fe700 1 -- 172.21.15.123:0/1433998870 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd7e801a840 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.850+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd7f4071010 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.851+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd7f4045ad0 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.851+0000 7fd7db7fe700 1 -- 172.21.15.123:0/1433998870 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fd7e800f070 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.852+0000 7fd7db7fe700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fd7e002fdb0 0x7fd7e0032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:20.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.852+0000 7fd7db7fe700 1 -- 172.21.15.123:0/1433998870 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fd7e8041170 con 0x7fd7f40fbcb0 2023-11-15T06:37:20.861 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.852+0000 7fd7f27fc700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fd7e002fdb0 0x7fd7e0032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:20.861 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.852+0000 7fd7f27fc700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fd7e002fdb0 0x7fd7e0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd7e40096a0 tx=0x7fd7e4006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:20.861 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:20.856+0000 7fd7db7fe700 1 -- 172.21.15.123:0/1433998870 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd7e8019950 con 0x7fd7f40fbcb0 2023-11-15T06:37:21.037 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.034+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7fd7f4071250 con 0x7fd7e002fdb0 2023-11-15T06:37:21.040 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.038+0000 7fd7db7fe700 1 -- 172.21.15.123:0/1433998870 <== mgr.14114 v2:172.21.15.123:6800/4164871557 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7fd7f4071250 con 0x7fd7e002fdb0 2023-11-15T06:37:21.041 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Scheduled crash update... 2023-11-15T06:37:21.043 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fd7e002fdb0 msgr2=0x7fd7e0032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:21.043 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fd7e002fdb0 0x7fd7e0032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7fd7e40096a0 tx=0x7fd7e4006b40).stop 2023-11-15T06:37:21.043 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 msgr2=0x7fd7f40725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:21.043 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40725e0 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7fd7e800b1d0 tx=0x7fd7e8006ee0).stop 2023-11-15T06:37:21.044 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 shutdown_connections 2023-11-15T06:37:21.044 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7fd7e002fdb0 0x7fd7e0032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:21.044 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 --2- 172.21.15.123:0/1433998870 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd7f40fbcb0 0x7fd7f40725e0 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:21.044 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.040+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 >> 172.21.15.123:0/1433998870 conn(0x7fd7f40f74b0 msgr2=0x7fd7f40f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:21.044 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.041+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 shutdown_connections 2023-11-15T06:37:21.045 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.041+0000 7fd7f96b1700 1 -- 172.21.15.123:0/1433998870 wait complete. 2023-11-15T06:37:21.399 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:20.085464+0000 mgr.smithi123.razjtn (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:21.399 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:20.085962+0000 mgr.smithi123.razjtn (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2023-11-15T06:37:21.399 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:20.088181+0000 mon.smithi123 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:21.400 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:21.038878+0000 mon.smithi123 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:Enabling mgr prometheus module... 2023-11-15T06:37:21.553 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2023-11-15T06:37:22.146 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.905+0000 7f002468f700 1 Processor -- start 2023-11-15T06:37:22.147 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.905+0000 7f002468f700 1 -- start start 2023-11-15T06:37:22.147 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f002468f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:22.147 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f002468f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c0fc750 con 0x7f001c0703b0 2023-11-15T06:37:22.147 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f002242b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:22.147 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f002242b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43112/0 (socket says 172.21.15.123:43112) 2023-11-15T06:37:22.148 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f002242b700 1 -- 172.21.15.123:0/3044029783 learned_addr learned my addr 172.21.15.123:0/3044029783 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:22.148 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f002242b700 1 -- 172.21.15.123:0/3044029783 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001c101250 con 0x7f001c0703b0 2023-11-15T06:37:22.148 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f002242b700 1 --2- 172.21.15.123:0/3044029783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c100d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f001800b5f0 tx=0x7f0018006ee0).ready entity=mon.0 client_cookie=76bbe17ef6b4edbf server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:22.148 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f0021429700 1 -- 172.21.15.123:0/3044029783 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f001800d600 con 0x7f001c0703b0 2023-11-15T06:37:22.148 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f0021429700 1 -- 172.21.15.123:0/3044029783 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f001800dbd0 con 0x7f001c0703b0 2023-11-15T06:37:22.149 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.906+0000 7f0021429700 1 -- 172.21.15.123:0/3044029783 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0018011c00 con 0x7f001c0703b0 2023-11-15T06:37:22.149 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 -- 172.21.15.123:0/3044029783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 msgr2=0x7f001c100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:22.149 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 --2- 172.21.15.123:0/3044029783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c100d60 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f001800b5f0 tx=0x7f0018006ee0).stop 2023-11-15T06:37:22.149 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 -- 172.21.15.123:0/3044029783 shutdown_connections 2023-11-15T06:37:22.149 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 --2- 172.21.15.123:0/3044029783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c100d60 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:22.150 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 -- 172.21.15.123:0/3044029783 >> 172.21.15.123:0/3044029783 conn(0x7f001c0f63c0 msgr2=0x7f001c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:22.150 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 -- 172.21.15.123:0/3044029783 shutdown_connections 2023-11-15T06:37:22.150 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 -- 172.21.15.123:0/3044029783 wait complete. 2023-11-15T06:37:22.150 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.907+0000 7f002468f700 1 Processor -- start 2023-11-15T06:37:22.150 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002468f700 1 -- start start 2023-11-15T06:37:22.151 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002468f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c10cc00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:22.151 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002468f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f001c10d0f0 con 0x7f001c0703b0 2023-11-15T06:37:22.151 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002242b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c10cc00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:22.151 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002242b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c10cc00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43122/0 (socket says 172.21.15.123:43122) 2023-11-15T06:37:22.151 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002242b700 1 -- 172.21.15.123:0/3642278535 learned_addr learned my addr 172.21.15.123:0/3642278535 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:22.152 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002242b700 1 -- 172.21.15.123:0/3642278535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f001800b200 con 0x7f001c0703b0 2023-11-15T06:37:22.152 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002242b700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c10cc00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f001800b190 tx=0x7f0018006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:22.152 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f001801b410 con 0x7f001c0703b0 2023-11-15T06:37:22.152 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f001801b9e0 con 0x7f001c0703b0 2023-11-15T06:37:22.152 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f001c10d2f0 con 0x7f001c0703b0 2023-11-15T06:37:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f001801a880 con 0x7f001c0703b0 2023-11-15T06:37:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.908+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f001c10d700 con 0x7f001c0703b0 2023-11-15T06:37:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.909+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f001800f070 con 0x7f001c0703b0 2023-11-15T06:37:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.909+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f001c045ad0 con 0x7f001c0703b0 2023-11-15T06:37:22.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.909+0000 7f000affd700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f000402fdb0 0x7f0004032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.909+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0018041220 con 0x7f001c0703b0 2023-11-15T06:37:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.909+0000 7f0021c2a700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f000402fdb0 0x7f0004032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.910+0000 7f0021c2a700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f000402fdb0 0x7f0004032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f000c0096a0 tx=0x7f000c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:21.914+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00180257e0 con 0x7f001c0703b0 2023-11-15T06:37:22.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:22.102+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7f001c101db0 con 0x7f001c0703b0 2023-11-15T06:37:22.199 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:22.195+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 7 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00180257e0 con 0x7f001c0703b0 2023-11-15T06:37:22.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:22 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:21.036039+0000 mgr.smithi123.razjtn (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:22.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:22 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:21.036644+0000 mgr.smithi123.razjtn (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2023-11-15T06:37:23.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.090+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f0018022d80 con 0x7f001c0703b0 2023-11-15T06:37:23.094 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.091+0000 7f000affd700 1 -- 172.21.15.123:0/3642278535 <== mon.0 v2:172.21.15.123:3300/0 9 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7f0018019300 con 0x7f001c0703b0 2023-11-15T06:37:23.096 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f000402fdb0 msgr2=0x7f0004032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:23.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f000402fdb0 0x7f0004032270 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7f000c0096a0 tx=0x7f000c006b40).stop 2023-11-15T06:37:23.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 msgr2=0x7f001c10cc00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:23.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c10cc00 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f001800b190 tx=0x7f0018006ee0).stop 2023-11-15T06:37:23.098 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 shutdown_connections 2023-11-15T06:37:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f000402fdb0 0x7f0004032270 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 --2- 172.21.15.123:0/3642278535 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f001c0703b0 0x7f001c10cc00 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 >> 172.21.15.123:0/3642278535 conn(0x7f001c0f63c0 msgr2=0x7f001c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 shutdown_connections 2023-11-15T06:37:23.099 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.093+0000 7f002468f700 1 -- 172.21.15.123:0/3642278535 wait complete. 2023-11-15T06:37:23.132 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:22.104721+0000 mon.smithi123 (mon.0) 59 : audit [INF] from='client.? 172.21.15.123:0/3642278535' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2023-11-15T06:37:23.132 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:22.197200+0000 mon.smithi123 (mon.0) 60 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:23.132 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:22.411344+0000 mon.smithi123 (mon.0) 61 : audit [INF] from='mgr.14114 172.21.15.123:0/164827969' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:23.444 INFO:teuthology.orchestra.run.smithi123.stderr:Deploying prometheus service with default placement... 2023-11-15T06:37:23.445 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2023-11-15T06:37:23.824 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.820+0000 7f0a71675700 1 Processor -- start 2023-11-15T06:37:23.828 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.820+0000 7f0a71675700 1 -- start start 2023-11-15T06:37:23.828 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a71675700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c0fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:23.828 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a71675700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c0fee30 con 0x7f0a6c0703d0 2023-11-15T06:37:23.828 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a6affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c0fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:23.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a6affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c0fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43138/0 (socket says 172.21.15.123:43138) 2023-11-15T06:37:23.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a6affd700 1 -- 172.21.15.123:0/4068166094 learned_addr learned my addr 172.21.15.123:0/4068166094 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:23.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a6affd700 1 -- 172.21.15.123:0/4068166094 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c0fef70 con 0x7f0a6c0703d0 2023-11-15T06:37:23.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a6affd700 1 --2- 172.21.15.123:0/4068166094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c0fe940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f0a5c00b5f0 tx=0x7f0a5c006ee0).ready entity=mon.0 client_cookie=54fcb140dfc0dd90 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:23.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.821+0000 7f0a69ffb700 1 -- 172.21.15.123:0/4068166094 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a5c00d600 con 0x7f0a6c0703d0 2023-11-15T06:37:23.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a69ffb700 1 -- 172.21.15.123:0/4068166094 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a5c00dc10 con 0x7f0a6c0703d0 2023-11-15T06:37:23.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a69ffb700 1 -- 172.21.15.123:0/4068166094 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a5c011cd0 con 0x7f0a6c0703d0 2023-11-15T06:37:23.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a71675700 1 -- 172.21.15.123:0/4068166094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 msgr2=0x7f0a6c0fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:23.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a71675700 1 --2- 172.21.15.123:0/4068166094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c0fe940 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f0a5c00b5f0 tx=0x7f0a5c006ee0).stop 2023-11-15T06:37:23.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a71675700 1 -- 172.21.15.123:0/4068166094 shutdown_connections 2023-11-15T06:37:23.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a71675700 1 --2- 172.21.15.123:0/4068166094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c0fe940 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:23.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a71675700 1 -- 172.21.15.123:0/4068166094 >> 172.21.15.123:0/4068166094 conn(0x7f0a6c0f63c0 msgr2=0x7f0a6c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:23.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a71675700 1 -- 172.21.15.123:0/4068166094 shutdown_connections 2023-11-15T06:37:23.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.822+0000 7f0a71675700 1 -- 172.21.15.123:0/4068166094 wait complete. 2023-11-15T06:37:23.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.823+0000 7f0a71675700 1 Processor -- start 2023-11-15T06:37:23.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.823+0000 7f0a71675700 1 -- start start 2023-11-15T06:37:23.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.823+0000 7f0a71675700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c10aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:23.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.823+0000 7f0a71675700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a6c10af30 con 0x7f0a6c0703d0 2023-11-15T06:37:23.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.823+0000 7f0a6affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c10aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:23.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.823+0000 7f0a6affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c10aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43140/0 (socket says 172.21.15.123:43140) 2023-11-15T06:37:23.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.823+0000 7f0a6affd700 1 -- 172.21.15.123:0/236426524 learned_addr learned my addr 172.21.15.123:0/236426524 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:23.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.824+0000 7f0a6affd700 1 -- 172.21.15.123:0/236426524 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a5c00b200 con 0x7f0a6c0703d0 2023-11-15T06:37:23.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.824+0000 7f0a6affd700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c10aa40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f0a5c006a40 tx=0x7f0a5c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:23.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.824+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a5c011c90 con 0x7f0a6c0703d0 2023-11-15T06:37:23.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.824+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a6c10b130 con 0x7f0a6c0703d0 2023-11-15T06:37:23.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.824+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a5c01b8c0 con 0x7f0a6c0703d0 2023-11-15T06:37:23.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.824+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a5c01a8c0 con 0x7f0a6c0703d0 2023-11-15T06:37:23.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.824+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a6c10b540 con 0x7f0a6c0703d0 2023-11-15T06:37:23.835 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.825+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7f0a5c00f070 con 0x7f0a6c0703d0 2023-11-15T06:37:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.825+0000 7f0a537fe700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.825+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a6c045ad0 con 0x7f0a6c0703d0 2023-11-15T06:37:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.825+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f0a5c040900 con 0x7f0a6c0703d0 2023-11-15T06:37:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.825+0000 7f0a6a7fc700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 msgr2=0x7f0a58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/4164871557 2023-11-15T06:37:23.836 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.825+0000 7f0a6a7fc700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:37:23.837 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:23.829+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a5c016070 con 0x7f0a6c0703d0 2023-11-15T06:37:24.017 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:24.014+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 --> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a6c104a10 con 0x7f0a5802fdb0 2023-11-15T06:37:24.029 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:24.026+0000 7f0a6a7fc700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 msgr2=0x7f0a58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/4164871557 2023-11-15T06:37:24.029 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:24.026+0000 7f0a6a7fc700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:37:24.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:24 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:23.093225+0000 mon.smithi123 (mon.0) 62 : audit [INF] from='client.? 172.21.15.123:0/3642278535' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2023-11-15T06:37:24.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:24 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:23.093274+0000 mon.smithi123 (mon.0) 63 : cluster [DBG] mgrmap e9: smithi123.razjtn(active, since 16s) 2023-11-15T06:37:24.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:24.427+0000 7f0a6a7fc700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 msgr2=0x7f0a58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/4164871557 2023-11-15T06:37:24.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:24.427+0000 7f0a6a7fc700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T06:37:25.231 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:25.228+0000 7f0a6a7fc700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 msgr2=0x7f0a58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/4164871557 2023-11-15T06:37:25.232 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:25.228+0000 7f0a6a7fc700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T06:37:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:26.829+0000 7f0a6a7fc700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 msgr2=0x7f0a58032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/4164871557 2023-11-15T06:37:26.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:26.829+0000 7f0a6a7fc700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T06:37:28.461 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:28.457+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7f0a5c024500 con 0x7f0a6c0703d0 2023-11-15T06:37:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:28.458+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 msgr2=0x7f0a58032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:37:28.462 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:28.458+0000 7f0a537fe700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:29.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:28.019816+0000 mon.smithi123 (mon.0) 64 : cluster [INF] Active manager daemon smithi123.razjtn restarted 2023-11-15T06:37:29.408 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:28.020241+0000 mon.smithi123 (mon.0) 65 : cluster [INF] Activating manager daemon smithi123.razjtn 2023-11-15T06:37:29.409 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:28.022643+0000 mon.smithi123 (mon.0) 66 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2023-11-15T06:37:29.409 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:28.460771+0000 mon.smithi123 (mon.0) 67 : cluster [DBG] mgrmap e10: smithi123.razjtn(active, starting, since 0.440639s) 2023-11-15T06:37:29.409 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.461624+0000 mon.smithi123 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2023-11-15T06:37:29.409 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.462204+0000 mon.smithi123 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mgr metadata", "who": "smithi123.razjtn", "id": "smithi123.razjtn"}]: dispatch 2023-11-15T06:37:29.410 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.463091+0000 mon.smithi123 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:37:29.410 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.463239+0000 mon.smithi123 (mon.0) 71 : audit [DBG] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:37:29.410 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.463367+0000 mon.smithi123 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:37:29.410 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:28.477093+0000 mon.smithi123 (mon.0) 73 : cluster [INF] Manager daemon smithi123.razjtn is now available 2023-11-15T06:37:29.410 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.573518+0000 mon.smithi123 (mon.0) 74 : audit [DBG] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:29.411 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.575363+0000 mon.smithi123 (mon.0) 75 : audit [DBG] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:29.411 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.588583+0000 mon.smithi123 (mon.0) 76 : audit [INF] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:37:29.411 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:28.591506+0000 mon.smithi123 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:37:29.463 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.459+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f0a5c022c00 con 0x7f0a6c0703d0 2023-11-15T06:37:29.463 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.459+0000 7f0a537fe700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0a58033c20 0x7f0a58036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:29.463 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.459+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 --> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7f0a6c104a10 con 0x7f0a58033c20 2023-11-15T06:37:29.464 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.459+0000 7f0a6a7fc700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0a58033c20 0x7f0a58036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:29.464 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.460+0000 7f0a6a7fc700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0a58033c20 0x7f0a58036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f0a600036d0 tx=0x7f0a60006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:29.467 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.464+0000 7f0a537fe700 1 -- 172.21.15.123:0/236426524 <== mgr.14142 v2:172.21.15.123:6800/1692115469 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7f0a6c104a10 con 0x7f0a58033c20 2023-11-15T06:37:29.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Scheduled prometheus update... 2023-11-15T06:37:29.470 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0a58033c20 msgr2=0x7f0a58036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:29.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0a58033c20 0x7f0a58036010 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f0a600036d0 tx=0x7f0a60006cb0).stop 2023-11-15T06:37:29.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 msgr2=0x7f0a6c10aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:29.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c10aa40 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7f0a5c006a40 tx=0x7f0a5c006ee0).stop 2023-11-15T06:37:29.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 shutdown_connections 2023-11-15T06:37:29.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0a58033c20 0x7f0a58036010 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:29.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a6c0703d0 0x7f0a6c10aa40 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:29.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 --2- 172.21.15.123:0/236426524 >> [v2:172.21.15.123:6800/4164871557,v1:172.21.15.123:6801/4164871557] conn(0x7f0a5802fdb0 0x7f0a58032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:29.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 >> 172.21.15.123:0/236426524 conn(0x7f0a6c0f63c0 msgr2=0x7f0a6c0f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:29.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 shutdown_connections 2023-11-15T06:37:29.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:29.466+0000 7f0a71675700 1 -- 172.21.15.123:0/236426524 wait complete. 2023-11-15T06:37:29.877 INFO:teuthology.orchestra.run.smithi123.stderr:Deploying grafana service with default placement... 2023-11-15T06:37:29.878 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2023-11-15T06:37:30.398 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.390+0000 7f0cf3b3d700 1 Processor -- start 2023-11-15T06:37:30.399 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.390+0000 7f0cf3b3d700 1 -- start start 2023-11-15T06:37:30.399 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.390+0000 7f0cf3b3d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec0fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:30.399 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.390+0000 7f0cf3b3d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec0fc5b0 con 0x7f0cec0fbca0 2023-11-15T06:37:30.399 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf18d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec0fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:30.399 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf18d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec0fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37974/0 (socket says 172.21.15.123:37974) 2023-11-15T06:37:30.400 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf18d9700 1 -- 172.21.15.123:0/1079096664 learned_addr learned my addr 172.21.15.123:0/1079096664 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:30.400 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf18d9700 1 -- 172.21.15.123:0/1079096664 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cec0fc6f0 con 0x7f0cec0fbca0 2023-11-15T06:37:30.400 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf18d9700 1 --2- 172.21.15.123:0/1079096664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec0fc0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0cdc00b5f0 tx=0x7f0cdc006ee0).ready entity=mon.0 client_cookie=30d61a242d44afcc server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:30.400 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf08d7700 1 -- 172.21.15.123:0/1079096664 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc00d600 con 0x7f0cec0fbca0 2023-11-15T06:37:30.401 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf08d7700 1 -- 172.21.15.123:0/1079096664 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cdc00dc10 con 0x7f0cec0fbca0 2023-11-15T06:37:30.401 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.391+0000 7f0cf08d7700 1 -- 172.21.15.123:0/1079096664 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc011cd0 con 0x7f0cec0fbca0 2023-11-15T06:37:30.401 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/1079096664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 msgr2=0x7f0cec0fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:30.402 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 --2- 172.21.15.123:0/1079096664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec0fc0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f0cdc00b5f0 tx=0x7f0cdc006ee0).stop 2023-11-15T06:37:30.402 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/1079096664 shutdown_connections 2023-11-15T06:37:30.402 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 --2- 172.21.15.123:0/1079096664 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec0fc0c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:30.402 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/1079096664 >> 172.21.15.123:0/1079096664 conn(0x7f0cec0f74c0 msgr2=0x7f0cec0f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:30.402 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/1079096664 shutdown_connections 2023-11-15T06:37:30.403 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/1079096664 wait complete. 2023-11-15T06:37:30.403 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 Processor -- start 2023-11-15T06:37:30.403 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.392+0000 7f0cf3b3d700 1 -- start start 2023-11-15T06:37:30.403 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf3b3d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec10cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:30.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf3b3d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0cec10d230 con 0x7f0cec0fbca0 2023-11-15T06:37:30.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf18d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec10cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:30.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf18d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec10cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37990/0 (socket says 172.21.15.123:37990) 2023-11-15T06:37:30.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf18d9700 1 -- 172.21.15.123:0/3705912942 learned_addr learned my addr 172.21.15.123:0/3705912942 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:30.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf18d9700 1 -- 172.21.15.123:0/3705912942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0cdc00b200 con 0x7f0cec0fbca0 2023-11-15T06:37:30.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf18d9700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec10cd40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0cdc00dfb0 tx=0x7f0cdc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:30.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc011df0 con 0x7f0cec0fbca0 2023-11-15T06:37:30.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0cdc01b8c0 con 0x7f0cec0fbca0 2023-11-15T06:37:30.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0cec10d430 con 0x7f0cec0fbca0 2023-11-15T06:37:30.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0cdc01a8c0 con 0x7f0cec0fbca0 2023-11-15T06:37:30.406 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.393+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0cec10d840 con 0x7f0cec0fbca0 2023-11-15T06:37:30.406 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.394+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f0cdc00f070 con 0x7f0cec0fbca0 2023-11-15T06:37:30.406 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.394+0000 7f0cda7fc700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0cd402fd50 0x7f0cd4032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:30.406 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.394+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f0cdc040f60 con 0x7f0cec0fbca0 2023-11-15T06:37:30.406 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.394+0000 7f0cf10d8700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0cd402fd50 0x7f0cd4032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:30.407 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.394+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0cec0fda20 con 0x7f0cec0fbca0 2023-11-15T06:37:30.407 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.395+0000 7f0cf10d8700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0cd402fd50 0x7f0cd4032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0ce80096a0 tx=0x7f0ce8006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:30.407 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.399+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0cdc016080 con 0x7f0cec0fbca0 2023-11-15T06:37:30.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.464+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f0cdc01ba30 con 0x7f0cec0fbca0 2023-11-15T06:37:30.585 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.582+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 --> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f0cec0fdbe0 con 0x7f0cd402fd50 2023-11-15T06:37:30.589 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.586+0000 7f0cda7fc700 1 -- 172.21.15.123:0/3705912942 <== mgr.14142 v2:172.21.15.123:6800/1692115469 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f0cec0fdbe0 con 0x7f0cd402fd50 2023-11-15T06:37:30.589 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Scheduled grafana update... 2023-11-15T06:37:30.591 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0cd402fd50 msgr2=0x7f0cd4032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:30.592 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0cd402fd50 0x7f0cd4032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f0ce80096a0 tx=0x7f0ce8006b40).stop 2023-11-15T06:37:30.592 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 msgr2=0x7f0cec10cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:30.592 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec10cd40 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f0cdc00dfb0 tx=0x7f0cdc006ee0).stop 2023-11-15T06:37:30.592 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 shutdown_connections 2023-11-15T06:37:30.592 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f0cd402fd50 0x7f0cd4032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:30.593 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 --2- 172.21.15.123:0/3705912942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0cec0fbca0 0x7f0cec10cd40 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:30.593 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 >> 172.21.15.123:0/3705912942 conn(0x7f0cec0f74c0 msgr2=0x7f0cec0f8f70 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:30.593 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.587+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 shutdown_connections 2023-11-15T06:37:30.593 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:30.588+0000 7f0cf3b3d700 1 -- 172.21.15.123:0/3705912942 wait complete. 2023-11-15T06:37:30.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:30 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:29.451994+0000 mon.smithi123 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:30.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:30 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:29.462800+0000 mon.smithi123 (mon.0) 79 : cluster [DBG] mgrmap e11: smithi123.razjtn(active, since 1.44267s) 2023-11-15T06:37:30.657 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:30 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:29.463857+0000 mgr.smithi123.razjtn (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2023-11-15T06:37:30.657 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:30 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:29.465926+0000 mon.smithi123 (mon.0) 80 : audit [INF] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:30.927 INFO:teuthology.orchestra.run.smithi123.stderr:Deploying node-exporter service with default placement... 2023-11-15T06:37:30.928 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2023-11-15T06:37:31.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.277+0000 7f3085713700 1 Processor -- start 2023-11-15T06:37:31.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.277+0000 7f3085713700 1 -- start start 2023-11-15T06:37:31.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.277+0000 7f3085713700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f30800fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:31.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.277+0000 7f3085713700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3080070770 con 0x7f3080070350 2023-11-15T06:37:31.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f30800fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:31.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f30800fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:37998/0 (socket says 172.21.15.123:37998) 2023-11-15T06:37:31.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307effd700 1 -- 172.21.15.123:0/927229168 learned_addr learned my addr 172.21.15.123:0/927229168 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:31.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307effd700 1 -- 172.21.15.123:0/927229168 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30800feeb0 con 0x7f3080070350 2023-11-15T06:37:31.284 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307effd700 1 --2- 172.21.15.123:0/927229168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f30800fe9c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f307400b5f0 tx=0x7f3074006ee0).ready entity=mon.0 client_cookie=d5b567491a4da1cd server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:31.284 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307dffb700 1 -- 172.21.15.123:0/927229168 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307400d600 con 0x7f3080070350 2023-11-15T06:37:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307dffb700 1 -- 172.21.15.123:0/927229168 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f307400dc10 con 0x7f3080070350 2023-11-15T06:37:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.278+0000 7f307dffb700 1 -- 172.21.15.123:0/927229168 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3074011cd0 con 0x7f3080070350 2023-11-15T06:37:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 -- 172.21.15.123:0/927229168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 msgr2=0x7f30800fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:31.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 --2- 172.21.15.123:0/927229168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f30800fe9c0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f307400b5f0 tx=0x7f3074006ee0).stop 2023-11-15T06:37:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 -- 172.21.15.123:0/927229168 shutdown_connections 2023-11-15T06:37:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 --2- 172.21.15.123:0/927229168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f30800fe9c0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 -- 172.21.15.123:0/927229168 >> 172.21.15.123:0/927229168 conn(0x7f30800f63c0 msgr2=0x7f30800f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 -- 172.21.15.123:0/927229168 shutdown_connections 2023-11-15T06:37:31.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 -- 172.21.15.123:0/927229168 wait complete. 2023-11-15T06:37:31.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 Processor -- start 2023-11-15T06:37:31.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.279+0000 7f3085713700 1 -- start start 2023-11-15T06:37:31.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f3085713700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f308010a9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:31.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f3085713700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f308010ae90 con 0x7f3080070350 2023-11-15T06:37:31.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f307effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f308010a9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:31.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f307effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f308010a9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38000/0 (socket says 172.21.15.123:38000) 2023-11-15T06:37:31.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f307effd700 1 -- 172.21.15.123:0/4179079075 learned_addr learned my addr 172.21.15.123:0/4179079075 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:31.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f307effd700 1 -- 172.21.15.123:0/4179079075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f307400b200 con 0x7f3080070350 2023-11-15T06:37:31.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f307effd700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f308010a9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f307400b1d0 tx=0x7f3074006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:31.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f30677fe700 1 -- 172.21.15.123:0/4179079075 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3074011df0 con 0x7f3080070350 2023-11-15T06:37:31.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f308010b090 con 0x7f3080070350 2023-11-15T06:37:31.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f30677fe700 1 -- 172.21.15.123:0/4179079075 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f307401b8c0 con 0x7f3080070350 2023-11-15T06:37:31.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f30677fe700 1 -- 172.21.15.123:0/4179079075 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f307401a8a0 con 0x7f3080070350 2023-11-15T06:37:31.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.280+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f308010b4a0 con 0x7f3080070350 2023-11-15T06:37:31.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.281+0000 7f30677fe700 1 -- 172.21.15.123:0/4179079075 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f307400f070 con 0x7f3080070350 2023-11-15T06:37:31.290 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.281+0000 7f30677fe700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f306c02fe60 0x7f306c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:31.291 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.281+0000 7f30677fe700 1 -- 172.21.15.123:0/4179079075 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f307401ba30 con 0x7f3080070350 2023-11-15T06:37:31.291 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.281+0000 7f307e7fc700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f306c02fe60 0x7f306c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:31.291 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.281+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3080059100 con 0x7f3080070350 2023-11-15T06:37:31.291 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.282+0000 7f307e7fc700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f306c02fe60 0x7f306c032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f30700096a0 tx=0x7f3070006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:31.292 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.286+0000 7f30677fe700 1 -- 172.21.15.123:0/4179079075 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3074016070 con 0x7f3080070350 2023-11-15T06:37:31.464 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.460+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 --> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7f30801048d0 con 0x7f306c02fe60 2023-11-15T06:37:31.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.464+0000 7f30677fe700 1 -- 172.21.15.123:0/4179079075 <== mgr.14142 v2:172.21.15.123:6800/1692115469 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7f30801048d0 con 0x7f306c02fe60 2023-11-15T06:37:31.468 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2023-11-15T06:37:31.470 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f306c02fe60 msgr2=0x7f306c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:31.470 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f306c02fe60 0x7f306c032320 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f30700096a0 tx=0x7f3070006b40).stop 2023-11-15T06:37:31.470 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 msgr2=0x7f308010a9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:31.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f308010a9a0 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f307400b1d0 tx=0x7f3074006ee0).stop 2023-11-15T06:37:31.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 shutdown_connections 2023-11-15T06:37:31.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f306c02fe60 0x7f306c032320 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:31.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 --2- 172.21.15.123:0/4179079075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3080070350 0x7f308010a9a0 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:31.471 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 >> 172.21.15.123:0/4179079075 conn(0x7f30800f63c0 msgr2=0x7f30800f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:31.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 shutdown_connections 2023-11-15T06:37:31.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:31.466+0000 7f3085713700 1 -- 172.21.15.123:0/4179079075 wait complete. 2023-11-15T06:37:31.762 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:30.467267+0000 mon.smithi123 (mon.0) 81 : cluster [DBG] mgrmap e12: smithi123.razjtn(active, since 2s) 2023-11-15T06:37:31.762 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:30.584601+0000 mgr.smithi123.razjtn (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:31.762 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:30.585147+0000 mgr.smithi123.razjtn (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2023-11-15T06:37:31.763 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:30.587503+0000 mon.smithi123 (mon.0) 82 : audit [INF] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:31.827 INFO:teuthology.orchestra.run.smithi123.stderr:Deploying alertmanager service with default placement... 2023-11-15T06:37:31.827 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2023-11-15T06:37:32.216 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.210+0000 7fdb1bfff700 1 Processor -- start 2023-11-15T06:37:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.210+0000 7fdb1bfff700 1 -- start start 2023-11-15T06:37:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.210+0000 7fdb1bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c0fe810 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.210+0000 7fdb1bfff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c070770 con 0x7fdb1c070350 2023-11-15T06:37:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.210+0000 7fdb1affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c0fe810 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:32.217 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.210+0000 7fdb1affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c0fe810 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38010/0 (socket says 172.21.15.123:38010) 2023-11-15T06:37:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.210+0000 7fdb1affd700 1 -- 172.21.15.123:0/3995132536 learned_addr learned my addr 172.21.15.123:0/3995132536 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1affd700 1 -- 172.21.15.123:0/3995132536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1c0fed00 con 0x7fdb1c070350 2023-11-15T06:37:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1affd700 1 --2- 172.21.15.123:0/3995132536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c0fe810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fdb1000b5f0 tx=0x7fdb10006ee0).ready entity=mon.0 client_cookie=9f672e66e0948dd3 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:32.218 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb19ffb700 1 -- 172.21.15.123:0/3995132536 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb1000d600 con 0x7fdb1c070350 2023-11-15T06:37:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb19ffb700 1 -- 172.21.15.123:0/3995132536 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb1000dc10 con 0x7fdb1c070350 2023-11-15T06:37:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb19ffb700 1 -- 172.21.15.123:0/3995132536 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb10011cd0 con 0x7fdb1c070350 2023-11-15T06:37:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3995132536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 msgr2=0x7fdb1c0fe810 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1bfff700 1 --2- 172.21.15.123:0/3995132536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c0fe810 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7fdb1000b5f0 tx=0x7fdb10006ee0).stop 2023-11-15T06:37:32.219 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3995132536 shutdown_connections 2023-11-15T06:37:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1bfff700 1 --2- 172.21.15.123:0/3995132536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c0fe810 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3995132536 >> 172.21.15.123:0/3995132536 conn(0x7fdb1c0f6210 msgr2=0x7fdb1c0f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.211+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3995132536 shutdown_connections 2023-11-15T06:37:32.220 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.212+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3995132536 wait complete. 2023-11-15T06:37:32.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.212+0000 7fdb1bfff700 1 Processor -- start 2023-11-15T06:37:32.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.212+0000 7fdb1bfff700 1 -- start start 2023-11-15T06:37:32.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c10a860 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:32.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1bfff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c10ad50 con 0x7fdb1c070350 2023-11-15T06:37:32.221 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c10a860 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:32.222 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c10a860 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38012/0 (socket says 172.21.15.123:38012) 2023-11-15T06:37:32.222 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1affd700 1 -- 172.21.15.123:0/3360660601 learned_addr learned my addr 172.21.15.123:0/3360660601 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:32.223 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1affd700 1 -- 172.21.15.123:0/3360660601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb1000b200 con 0x7fdb1c070350 2023-11-15T06:37:32.223 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1affd700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c10a860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdb1000dfb0 tx=0x7fdb10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:32.224 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb037fe700 1 -- 172.21.15.123:0/3360660601 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb1001b410 con 0x7fdb1c070350 2023-11-15T06:37:32.224 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1c10af50 con 0x7fdb1c070350 2023-11-15T06:37:32.224 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb037fe700 1 -- 172.21.15.123:0/3360660601 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb1001ba20 con 0x7fdb1c070350 2023-11-15T06:37:32.224 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb037fe700 1 -- 172.21.15.123:0/3360660601 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb1001a9e0 con 0x7fdb1c070350 2023-11-15T06:37:32.224 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.213+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1c10b360 con 0x7fdb1c070350 2023-11-15T06:37:32.225 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.214+0000 7fdb037fe700 1 -- 172.21.15.123:0/3360660601 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fdb1000f070 con 0x7fdb1c070350 2023-11-15T06:37:32.225 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.214+0000 7fdb037fe700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fdb0802fe60 0x7fdb08032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:32.225 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.214+0000 7fdb037fe700 1 -- 172.21.15.123:0/3360660601 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdb10041190 con 0x7fdb1c070350 2023-11-15T06:37:32.225 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.215+0000 7fdb1a7fc700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fdb0802fe60 0x7fdb08032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:32.225 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.215+0000 7fdb1a7fc700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fdb0802fe60 0x7fdb08032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdb0c0096a0 tx=0x7fdb0c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:32.226 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.215+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1c059100 con 0x7fdb1c070350 2023-11-15T06:37:32.226 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.220+0000 7fdb037fe700 1 -- 172.21.15.123:0/3360660601 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb10016070 con 0x7fdb1c070350 2023-11-15T06:37:32.398 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.395+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 --> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7fdb1c1046a0 con 0x7fdb0802fe60 2023-11-15T06:37:32.402 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.398+0000 7fdb037fe700 1 -- 172.21.15.123:0/3360660601 <== mgr.14142 v2:172.21.15.123:6800/1692115469 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7fdb1c1046a0 con 0x7fdb0802fe60 2023-11-15T06:37:32.402 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2023-11-15T06:37:32.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fdb0802fe60 msgr2=0x7fdb08032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:32.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fdb0802fe60 0x7fdb08032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fdb0c0096a0 tx=0x7fdb0c006b40).stop 2023-11-15T06:37:32.404 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 msgr2=0x7fdb1c10a860 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c10a860 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7fdb1000dfb0 tx=0x7fdb10006ee0).stop 2023-11-15T06:37:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 shutdown_connections 2023-11-15T06:37:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fdb0802fe60 0x7fdb08032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 --2- 172.21.15.123:0/3360660601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdb1c070350 0x7fdb1c10a860 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:32.405 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 >> 172.21.15.123:0/3360660601 conn(0x7fdb1c0f6210 msgr2=0x7fdb1c0f7db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:32.406 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 shutdown_connections 2023-11-15T06:37:32.407 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:32.400+0000 7fdb1bfff700 1 -- 172.21.15.123:0/3360660601 wait complete. 2023-11-15T06:37:32.695 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:31.462931+0000 mgr.smithi123.razjtn (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:32.695 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:31.463532+0000 mgr.smithi123.razjtn (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2023-11-15T06:37:32.696 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:31.465695+0000 mon.smithi123 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:32.696 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:32.400223+0000 mon.smithi123 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.123:0/1766503669' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:32.759 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2023-11-15T06:37:33.136 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.130+0000 7f7668b7e700 1 Processor -- start 2023-11-15T06:37:33.136 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.130+0000 7f7668b7e700 1 -- start start 2023-11-15T06:37:33.136 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f7668b7e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76640fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f7668b7e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76640fb490 con 0x7f76640fab80 2023-11-15T06:37:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76640fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76640fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38028/0 (socket says 172.21.15.123:38028) 2023-11-15T06:37:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766259c700 1 -- 172.21.15.123:0/3220477644 learned_addr learned my addr 172.21.15.123:0/3220477644 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:33.137 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766259c700 1 -- 172.21.15.123:0/3220477644 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76640fb5d0 con 0x7f76640fab80 2023-11-15T06:37:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766259c700 1 --2- 172.21.15.123:0/3220477644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76640fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f765400b5f0 tx=0x7f7654006ee0).ready entity=mon.0 client_cookie=fe168806fa24896 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766159a700 1 -- 172.21.15.123:0/3220477644 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765400d600 con 0x7f76640fab80 2023-11-15T06:37:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766159a700 1 -- 172.21.15.123:0/3220477644 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f765400dc10 con 0x7f76640fab80 2023-11-15T06:37:33.138 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.131+0000 7f766159a700 1 -- 172.21.15.123:0/3220477644 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7654011cd0 con 0x7f76640fab80 2023-11-15T06:37:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 -- 172.21.15.123:0/3220477644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 msgr2=0x7f76640fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 --2- 172.21.15.123:0/3220477644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76640fafa0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f765400b5f0 tx=0x7f7654006ee0).stop 2023-11-15T06:37:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 -- 172.21.15.123:0/3220477644 shutdown_connections 2023-11-15T06:37:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 --2- 172.21.15.123:0/3220477644 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76640fafa0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 -- 172.21.15.123:0/3220477644 >> 172.21.15.123:0/3220477644 conn(0x7f76640f63c0 msgr2=0x7f76640f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:33.139 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 -- 172.21.15.123:0/3220477644 shutdown_connections 2023-11-15T06:37:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 -- 172.21.15.123:0/3220477644 wait complete. 2023-11-15T06:37:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 Processor -- start 2023-11-15T06:37:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.132+0000 7f7668b7e700 1 -- start start 2023-11-15T06:37:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f7668b7e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76641088c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:33.140 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f7668b7e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7664108db0 con 0x7f76640fab80 2023-11-15T06:37:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f766259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76641088c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f766259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76641088c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38042/0 (socket says 172.21.15.123:38042) 2023-11-15T06:37:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f766259c700 1 -- 172.21.15.123:0/2025570319 learned_addr learned my addr 172.21.15.123:0/2025570319 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f766259c700 1 -- 172.21.15.123:0/2025570319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f765400b200 con 0x7f76640fab80 2023-11-15T06:37:33.141 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f766259c700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76641088c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f765400dfb0 tx=0x7f7654006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:33.142 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f765affd700 1 -- 172.21.15.123:0/2025570319 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7654011df0 con 0x7f76640fab80 2023-11-15T06:37:33.142 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f765affd700 1 -- 172.21.15.123:0/2025570319 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f765401b8c0 con 0x7f76640fab80 2023-11-15T06:37:33.142 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7664108fb0 con 0x7f76640fab80 2023-11-15T06:37:33.143 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f765affd700 1 -- 172.21.15.123:0/2025570319 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f765401a8c0 con 0x7f76640fab80 2023-11-15T06:37:33.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.133+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76641093c0 con 0x7f76640fab80 2023-11-15T06:37:33.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.134+0000 7f765affd700 1 -- 172.21.15.123:0/2025570319 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f765400f070 con 0x7f76640fab80 2023-11-15T06:37:33.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.134+0000 7f765affd700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f7644032100 0x7f76440345c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:33.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.134+0000 7f765affd700 1 -- 172.21.15.123:0/2025570319 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7654040fe0 con 0x7f76640fab80 2023-11-15T06:37:33.144 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.134+0000 7f7661d9b700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f7644032100 0x7f76440345c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:33.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.134+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7664059100 con 0x7f76640fab80 2023-11-15T06:37:33.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.135+0000 7f7661d9b700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f7644032100 0x7f76440345c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f764c0096a0 tx=0x7f764c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:33.145 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.139+0000 7f765affd700 1 -- 172.21.15.123:0/2025570319 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7654016070 con 0x7f76640fab80 2023-11-15T06:37:33.291 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.288+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f76641025d0 con 0x7f76640fab80 2023-11-15T06:37:33.294 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.291+0000 7f765affd700 1 -- 172.21.15.123:0/2025570319 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v9)=0 v9) v1 ==== 142+0+0 (secure 0 0 0) 0x7f7654024410 con 0x7f76640fab80 2023-11-15T06:37:33.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f7644032100 msgr2=0x7f76440345c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:33.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f7644032100 0x7f76440345c0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f764c0096a0 tx=0x7f764c006b40).stop 2023-11-15T06:37:33.296 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 msgr2=0x7f76641088c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:33.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76641088c0 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f765400dfb0 tx=0x7f7654006ee0).stop 2023-11-15T06:37:33.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 shutdown_connections 2023-11-15T06:37:33.297 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f7644032100 0x7f76440345c0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:33.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 --2- 172.21.15.123:0/2025570319 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76640fab80 0x7f76641088c0 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:33.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.293+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 >> 172.21.15.123:0/2025570319 conn(0x7f76640f63c0 msgr2=0x7f76640ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:33.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.295+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 shutdown_connections 2023-11-15T06:37:33.298 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.295+0000 7f7668b7e700 1 -- 172.21.15.123:0/2025570319 wait complete. 2023-11-15T06:37:33.578 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:32.397630+0000 mgr.smithi123.razjtn (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:33.578 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:32.398142+0000 mgr.smithi123.razjtn (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2023-11-15T06:37:33.579 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:33.292477+0000 mon.smithi123 (mon.0) 85 : audit [INF] from='client.? 172.21.15.123:0/2025570319' entity='client.admin' 2023-11-15T06:37:33.643 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2023-11-15T06:37:33.992 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.988+0000 7fcc18e0a700 1 Processor -- start 2023-11-15T06:37:33.993 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.988+0000 7fcc18e0a700 1 -- start start 2023-11-15T06:37:33.993 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.988+0000 7fcc18e0a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:33.993 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.988+0000 7fcc18e0a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140fedd0 con 0x7fcc14070430 2023-11-15T06:37:33.993 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:33.993 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38056/0 (socket says 172.21.15.123:38056) 2023-11-15T06:37:33.994 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1259c700 1 -- 172.21.15.123:0/3343392848 learned_addr learned my addr 172.21.15.123:0/3343392848 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:33.994 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1259c700 1 -- 172.21.15.123:0/3343392848 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc140fef10 con 0x7fcc14070430 2023-11-15T06:37:33.994 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1259c700 1 --2- 172.21.15.123:0/3343392848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fe8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fcc0800b5f0 tx=0x7fcc08006ee0).ready entity=mon.0 client_cookie=5cbe6dc9c13b801f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:33.994 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1159a700 1 -- 172.21.15.123:0/3343392848 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc0800d600 con 0x7fcc14070430 2023-11-15T06:37:33.994 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1159a700 1 -- 172.21.15.123:0/3343392848 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc0800dc10 con 0x7fcc14070430 2023-11-15T06:37:33.995 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.989+0000 7fcc1159a700 1 -- 172.21.15.123:0/3343392848 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc08011cd0 con 0x7fcc14070430 2023-11-15T06:37:33.995 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3343392848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 msgr2=0x7fcc140fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:33.995 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 --2- 172.21.15.123:0/3343392848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fe8e0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7fcc0800b5f0 tx=0x7fcc08006ee0).stop 2023-11-15T06:37:33.995 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3343392848 shutdown_connections 2023-11-15T06:37:33.995 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 --2- 172.21.15.123:0/3343392848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fe8e0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:33.996 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3343392848 >> 172.21.15.123:0/3343392848 conn(0x7fcc140f63c0 msgr2=0x7fcc140f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:33.996 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3343392848 shutdown_connections 2023-11-15T06:37:33.997 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3343392848 wait complete. 2023-11-15T06:37:33.997 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 Processor -- start 2023-11-15T06:37:33.997 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 -- start start 2023-11-15T06:37:33.997 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:33.997 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc18e0a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc140fa7e0 con 0x7fcc14070430 2023-11-15T06:37:33.998 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc1259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:33.998 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc1259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38064/0 (socket says 172.21.15.123:38064) 2023-11-15T06:37:33.998 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.990+0000 7fcc1259c700 1 -- 172.21.15.123:0/3921200373 learned_addr learned my addr 172.21.15.123:0/3921200373 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:33.998 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.991+0000 7fcc1259c700 1 -- 172.21.15.123:0/3921200373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc0800b200 con 0x7fcc14070430 2023-11-15T06:37:33.998 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.991+0000 7fcc1259c700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fc360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fcc08006800 tx=0x7fcc08006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.991+0000 7fcc02ffd700 1 -- 172.21.15.123:0/3921200373 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc08011df0 con 0x7fcc14070430 2023-11-15T06:37:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.991+0000 7fcc02ffd700 1 -- 172.21.15.123:0/3921200373 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc0801b8c0 con 0x7fcc14070430 2023-11-15T06:37:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.991+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc140fa980 con 0x7fcc14070430 2023-11-15T06:37:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.991+0000 7fcc02ffd700 1 -- 172.21.15.123:0/3921200373 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc0801a830 con 0x7fcc14070430 2023-11-15T06:37:33.999 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.991+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc140fad90 con 0x7fcc14070430 2023-11-15T06:37:34.000 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.992+0000 7fcc02ffd700 1 -- 172.21.15.123:0/3921200373 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7fcc0800f070 con 0x7fcc14070430 2023-11-15T06:37:34.000 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.992+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc14045ad0 con 0x7fcc14070430 2023-11-15T06:37:34.000 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.992+0000 7fcc02ffd700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fcbf802fe60 0x7fcbf8032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:34.000 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.992+0000 7fcc02ffd700 1 -- 172.21.15.123:0/3921200373 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fcc08041000 con 0x7fcc14070430 2023-11-15T06:37:34.000 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.992+0000 7fcc11d9b700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fcbf802fe60 0x7fcbf8032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:34.001 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.992+0000 7fcc11d9b700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fcbf802fe60 0x7fcbf8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcc040096a0 tx=0x7fcc04006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:34.001 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:33.996+0000 7fcc02ffd700 1 -- 172.21.15.123:0/3921200373 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc08010be0 con 0x7fcc14070430 2023-11-15T06:37:34.147 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.145+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7fcc140ffc00 con 0x7fcc14070430 2023-11-15T06:37:34.150 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.148+0000 7fcc02ffd700 1 -- 172.21.15.123:0/3921200373 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7fcc08016070 con 0x7fcc14070430 2023-11-15T06:37:34.152 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fcbf802fe60 msgr2=0x7fcbf8032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:34.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fcbf802fe60 0x7fcbf8032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fcc040096a0 tx=0x7fcc04006b40).stop 2023-11-15T06:37:34.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 msgr2=0x7fcc140fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:34.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fc360 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fcc08006800 tx=0x7fcc08006ee0).stop 2023-11-15T06:37:34.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 shutdown_connections 2023-11-15T06:37:34.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fcbf802fe60 0x7fcbf8032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:34.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 --2- 172.21.15.123:0/3921200373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcc14070430 0x7fcc140fc360 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:34.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 >> 172.21.15.123:0/3921200373 conn(0x7fcc140f63c0 msgr2=0x7fcc140f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:34.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.149+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 shutdown_connections 2023-11-15T06:37:34.154 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.150+0000 7fcc18e0a700 1 -- 172.21.15.123:0/3921200373 wait complete. 2023-11-15T06:37:34.509 INFO:teuthology.orchestra.run.smithi123.stderr:Enabling the dashboard module... 2023-11-15T06:37:34.510 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2023-11-15T06:37:34.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.854+0000 7f5885984700 1 Processor -- start 2023-11-15T06:37:34.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.854+0000 7f5885984700 1 -- start start 2023-11-15T06:37:34.858 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.854+0000 7f5885984700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f58800fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:34.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.854+0000 7f5885984700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58800fede0 con 0x7f58800704a0 2023-11-15T06:37:34.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f58800fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:34.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f58800fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38080/0 (socket says 172.21.15.123:38080) 2023-11-15T06:37:34.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587effd700 1 -- 172.21.15.123:0/466143148 learned_addr learned my addr 172.21.15.123:0/466143148 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:34.859 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587effd700 1 -- 172.21.15.123:0/466143148 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f58800fef20 con 0x7f58800704a0 2023-11-15T06:37:34.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587effd700 1 --2- 172.21.15.123:0/466143148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f58800fe8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f587400b5f0 tx=0x7f5874006ee0).ready entity=mon.0 client_cookie=c124e3ba3535585f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:34.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587dffb700 1 -- 172.21.15.123:0/466143148 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f587400d600 con 0x7f58800704a0 2023-11-15T06:37:34.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587dffb700 1 -- 172.21.15.123:0/466143148 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f587400dc10 con 0x7f58800704a0 2023-11-15T06:37:34.860 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.855+0000 7f587dffb700 1 -- 172.21.15.123:0/466143148 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5874011cd0 con 0x7f58800704a0 2023-11-15T06:37:34.863 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 -- 172.21.15.123:0/466143148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 msgr2=0x7f58800fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:34.863 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 --2- 172.21.15.123:0/466143148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f58800fe8f0 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f587400b5f0 tx=0x7f5874006ee0).stop 2023-11-15T06:37:34.864 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 -- 172.21.15.123:0/466143148 shutdown_connections 2023-11-15T06:37:34.864 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 --2- 172.21.15.123:0/466143148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f58800fe8f0 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:34.864 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 -- 172.21.15.123:0/466143148 >> 172.21.15.123:0/466143148 conn(0x7f58800f63b0 msgr2=0x7f58800f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:34.864 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 -- 172.21.15.123:0/466143148 shutdown_connections 2023-11-15T06:37:34.864 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 -- 172.21.15.123:0/466143148 wait complete. 2023-11-15T06:37:34.865 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 Processor -- start 2023-11-15T06:37:34.865 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.856+0000 7f5885984700 1 -- start start 2023-11-15T06:37:34.865 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f5885984700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f588010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:34.865 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f5885984700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f588010af30 con 0x7f58800704a0 2023-11-15T06:37:34.866 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f587effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f588010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:34.866 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f587effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f588010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38084/0 (socket says 172.21.15.123:38084) 2023-11-15T06:37:34.866 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f587effd700 1 -- 172.21.15.123:0/2575836258 learned_addr learned my addr 172.21.15.123:0/2575836258 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:34.866 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f587effd700 1 -- 172.21.15.123:0/2575836258 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f587400b200 con 0x7f58800704a0 2023-11-15T06:37:34.866 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f587effd700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f588010aa40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5874006a40 tx=0x7f5874006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:34.867 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5874011c90 con 0x7f58800704a0 2023-11-15T06:37:34.867 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f588010b130 con 0x7f58800704a0 2023-11-15T06:37:34.867 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f587401b8c0 con 0x7f58800704a0 2023-11-15T06:37:34.867 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f587401a8c0 con 0x7f58800704a0 2023-11-15T06:37:34.867 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.857+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f588010b540 con 0x7f58800704a0 2023-11-15T06:37:34.868 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.858+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f587400f070 con 0x7f58800704a0 2023-11-15T06:37:34.868 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.858+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5880045ad0 con 0x7f58800704a0 2023-11-15T06:37:34.868 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.858+0000 7f58677fe700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f586c02fe60 0x7f586c032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:34.868 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.858+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5874040a10 con 0x7f58800704a0 2023-11-15T06:37:34.868 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.859+0000 7f587e7fc700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f586c02fe60 0x7f586c032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:34.869 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.859+0000 7f587e7fc700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f586c02fe60 0x7f586c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f58700096a0 tx=0x7f5870006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:34.869 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:34.863+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5874010930 con 0x7f58800704a0 2023-11-15T06:37:35.051 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.048+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f58800ffc10 con 0x7f58800704a0 2023-11-15T06:37:35.152 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.149+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f5874024550 con 0x7f58800704a0 2023-11-15T06:37:35.153 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.150+0000 7f58677fe700 1 -- 172.21.15.123:0/2575836258 <== mon.0 v2:172.21.15.123:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f5874022ce0 con 0x7f58800704a0 2023-11-15T06:37:35.155 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f586c02fe60 msgr2=0x7f586c032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:35.156 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f586c02fe60 0x7f586c032320 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f58700096a0 tx=0x7f5870006b40).stop 2023-11-15T06:37:35.156 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 msgr2=0x7f588010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:35.156 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f588010aa40 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5874006a40 tx=0x7f5874006ee0).stop 2023-11-15T06:37:35.156 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 shutdown_connections 2023-11-15T06:37:35.156 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f586c02fe60 0x7f586c032320 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:35.157 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 --2- 172.21.15.123:0/2575836258 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f58800704a0 0x7f588010aa40 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:35.157 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 >> 172.21.15.123:0/2575836258 conn(0x7f58800f63b0 msgr2=0x7f58800f7e60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:35.157 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.152+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 shutdown_connections 2023-11-15T06:37:35.157 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.153+0000 7f5885984700 1 -- 172.21.15.123:0/2575836258 wait complete. 2023-11-15T06:37:35.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:34.148732+0000 mon.smithi123 (mon.0) 86 : audit [INF] from='client.? 172.21.15.123:0/3921200373' entity='client.admin' 2023-11-15T06:37:35.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:35.049934+0000 mon.smithi123 (mon.0) 87 : audit [INF] from='client.? 172.21.15.123:0/2575836258' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2023-11-15T06:37:35.509 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2023-11-15T06:37:35.882 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.878+0000 7f224110c700 1 Processor -- start 2023-11-15T06:37:35.883 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f224110c700 1 -- start start 2023-11-15T06:37:35.883 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f224110c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c100d40 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:35.883 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f224110c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c0fc730 con 0x7f223c0703d0 2023-11-15T06:37:35.883 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f223ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c100d40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:35.884 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f223ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c100d40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38112/0 (socket says 172.21.15.123:38112) 2023-11-15T06:37:35.884 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f223ad9d700 1 -- 172.21.15.123:0/312015296 learned_addr learned my addr 172.21.15.123:0/312015296 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:35.884 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f223ad9d700 1 -- 172.21.15.123:0/312015296 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f223c101230 con 0x7f223c0703d0 2023-11-15T06:37:35.884 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f223ad9d700 1 --2- 172.21.15.123:0/312015296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c100d40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f222c00b5f0 tx=0x7f222c006ee0).ready entity=mon.0 client_cookie=eb29877b3a58dde8 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:35.884 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f2239d9b700 1 -- 172.21.15.123:0/312015296 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f222c00d600 con 0x7f223c0703d0 2023-11-15T06:37:35.885 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.879+0000 7f2239d9b700 1 -- 172.21.15.123:0/312015296 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f222c00dc10 con 0x7f223c0703d0 2023-11-15T06:37:35.885 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f2239d9b700 1 -- 172.21.15.123:0/312015296 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f222c011cd0 con 0x7f223c0703d0 2023-11-15T06:37:35.885 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f224110c700 1 -- 172.21.15.123:0/312015296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 msgr2=0x7f223c100d40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:35.885 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f224110c700 1 --2- 172.21.15.123:0/312015296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c100d40 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7f222c00b5f0 tx=0x7f222c006ee0).stop 2023-11-15T06:37:35.889 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f224110c700 1 -- 172.21.15.123:0/312015296 shutdown_connections 2023-11-15T06:37:35.889 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f224110c700 1 --2- 172.21.15.123:0/312015296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c100d40 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:35.889 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f224110c700 1 -- 172.21.15.123:0/312015296 >> 172.21.15.123:0/312015296 conn(0x7f223c0f63c0 msgr2=0x7f223c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:35.889 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f224110c700 1 -- 172.21.15.123:0/312015296 shutdown_connections 2023-11-15T06:37:35.890 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.880+0000 7f224110c700 1 -- 172.21.15.123:0/312015296 wait complete. 2023-11-15T06:37:35.890 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.881+0000 7f224110c700 1 Processor -- start 2023-11-15T06:37:35.890 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.881+0000 7f224110c700 1 -- start start 2023-11-15T06:37:35.890 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.881+0000 7f224110c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:35.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.881+0000 7f224110c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f223c10d1f0 con 0x7f223c0703d0 2023-11-15T06:37:35.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f223ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:35.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f223ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38128/0 (socket says 172.21.15.123:38128) 2023-11-15T06:37:35.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f223ad9d700 1 -- 172.21.15.123:0/2044834185 learned_addr learned my addr 172.21.15.123:0/2044834185 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:35.891 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f223ad9d700 1 -- 172.21.15.123:0/2044834185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f222c00b200 con 0x7f223c0703d0 2023-11-15T06:37:35.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f223ad9d700 1 --2- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c10cd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f222c006a40 tx=0x7f222c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:35.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f22237fe700 1 -- 172.21.15.123:0/2044834185 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f222c011c90 con 0x7f223c0703d0 2023-11-15T06:37:35.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f22237fe700 1 -- 172.21.15.123:0/2044834185 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f222c01b8c0 con 0x7f223c0703d0 2023-11-15T06:37:35.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f223c10d3f0 con 0x7f223c0703d0 2023-11-15T06:37:35.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f22237fe700 1 -- 172.21.15.123:0/2044834185 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f222c01a8c0 con 0x7f223c0703d0 2023-11-15T06:37:35.892 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.882+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f223c10d800 con 0x7f223c0703d0 2023-11-15T06:37:35.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.883+0000 7f22237fe700 1 -- 172.21.15.123:0/2044834185 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f222c00f070 con 0x7f223c0703d0 2023-11-15T06:37:35.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.883+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f223c045ad0 con 0x7f223c0703d0 2023-11-15T06:37:35.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.883+0000 7f22237fe700 1 --2- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f222802feb0 0x7f2228032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:35.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.883+0000 7f22237fe700 1 -- 172.21.15.123:0/2044834185 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f222c040b90 con 0x7f223c0703d0 2023-11-15T06:37:35.893 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.883+0000 7f223a59c700 1 -- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f222802feb0 msgr2=0x7f2228032370 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1692115469 2023-11-15T06:37:35.894 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.883+0000 7f223a59c700 1 --2- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f222802feb0 0x7f2228032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:37:35.894 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:35.888+0000 7f22237fe700 1 -- 172.21.15.123:0/2044834185 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f222c010540 con 0x7f223c0703d0 2023-11-15T06:37:36.084 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.081+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7f223c10e6a0 con 0x7f223c0703d0 2023-11-15T06:37:36.085 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.082+0000 7f22237fe700 1 -- 172.21.15.123:0/2044834185 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7f222c016070 con 0x7f223c0703d0 2023-11-15T06:37:36.085 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: { 2023-11-15T06:37:36.085 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "epoch": 13, 2023-11-15T06:37:36.085 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "available": true, 2023-11-15T06:37:36.086 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "active_name": "smithi123.razjtn", 2023-11-15T06:37:36.086 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "num_standby": 0 2023-11-15T06:37:36.087 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: } 2023-11-15T06:37:36.087 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f222802feb0 msgr2=0x7f2228032370 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:37:36.088 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 --2- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f222802feb0 0x7f2228032370 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:36.088 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 msgr2=0x7f223c10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:36.088 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 --2- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c10cd00 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f222c006a40 tx=0x7f222c006ee0).stop 2023-11-15T06:37:36.088 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 shutdown_connections 2023-11-15T06:37:36.088 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 --2- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7f222802feb0 0x7f2228032370 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:36.089 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 --2- 172.21.15.123:0/2044834185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f223c0703d0 0x7f223c10cd00 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:36.089 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 >> 172.21.15.123:0/2044834185 conn(0x7f223c0f63c0 msgr2=0x7f223c0f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:36.089 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 shutdown_connections 2023-11-15T06:37:36.089 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.083+0000 7f224110c700 1 -- 172.21.15.123:0/2044834185 wait complete. 2023-11-15T06:37:36.369 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:36 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:35.151386+0000 mon.smithi123 (mon.0) 88 : audit [INF] from='client.? 172.21.15.123:0/2575836258' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2023-11-15T06:37:36.370 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:36 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:35.151437+0000 mon.smithi123 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi123.razjtn(active, since 7s) 2023-11-15T06:37:36.370 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:36 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:36.082616+0000 mon.smithi123 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.123:0/2044834185' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2023-11-15T06:37:36.436 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for the mgr to restart... 2023-11-15T06:37:36.436 INFO:teuthology.orchestra.run.smithi123.stderr:Waiting for mgr epoch 13... 2023-11-15T06:37:36.437 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2023-11-15T06:37:36.828 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.821+0000 7fd316b0a700 1 Processor -- start 2023-11-15T06:37:36.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd316b0a700 1 -- start start 2023-11-15T06:37:36.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd316b0a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd310100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:36.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd316b0a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3100fc750 con 0x7fd3100703b0 2023-11-15T06:37:36.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd3148a6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd310100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:36.829 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd3148a6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd310100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38130/0 (socket says 172.21.15.123:38130) 2023-11-15T06:37:36.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd3148a6700 1 -- 172.21.15.123:0/3058435620 learned_addr learned my addr 172.21.15.123:0/3058435620 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:36.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd3148a6700 1 -- 172.21.15.123:0/3058435620 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd310101250 con 0x7fd3100703b0 2023-11-15T06:37:36.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd3148a6700 1 --2- 172.21.15.123:0/3058435620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd310100d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd30400b5f0 tx=0x7fd304006ee0).ready entity=mon.0 client_cookie=20bc033d2c01cb82 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:36.830 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd30f7fe700 1 -- 172.21.15.123:0/3058435620 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30400d600 con 0x7fd3100703b0 2023-11-15T06:37:36.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd30f7fe700 1 -- 172.21.15.123:0/3058435620 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd30400dc10 con 0x7fd3100703b0 2023-11-15T06:37:36.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.822+0000 7fd30f7fe700 1 -- 172.21.15.123:0/3058435620 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd304011cd0 con 0x7fd3100703b0 2023-11-15T06:37:36.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.823+0000 7fd316b0a700 1 -- 172.21.15.123:0/3058435620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 msgr2=0x7fd310100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:36.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.823+0000 7fd316b0a700 1 --2- 172.21.15.123:0/3058435620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd310100d60 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fd30400b5f0 tx=0x7fd304006ee0).stop 2023-11-15T06:37:36.831 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.823+0000 7fd316b0a700 1 -- 172.21.15.123:0/3058435620 shutdown_connections 2023-11-15T06:37:36.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.823+0000 7fd316b0a700 1 --2- 172.21.15.123:0/3058435620 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd310100d60 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:36.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.823+0000 7fd316b0a700 1 -- 172.21.15.123:0/3058435620 >> 172.21.15.123:0/3058435620 conn(0x7fd3100f63c0 msgr2=0x7fd3100f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:36.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.823+0000 7fd316b0a700 1 -- 172.21.15.123:0/3058435620 shutdown_connections 2023-11-15T06:37:36.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.823+0000 7fd316b0a700 1 -- 172.21.15.123:0/3058435620 wait complete. 2023-11-15T06:37:36.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.824+0000 7fd316b0a700 1 Processor -- start 2023-11-15T06:37:36.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.824+0000 7fd316b0a700 1 -- start start 2023-11-15T06:37:36.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.824+0000 7fd316b0a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd31010cce0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:36.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.824+0000 7fd316b0a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd31010d1d0 con 0x7fd3100703b0 2023-11-15T06:37:36.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.824+0000 7fd3148a6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd31010cce0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:36.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.824+0000 7fd3148a6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd31010cce0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:38134/0 (socket says 172.21.15.123:38134) 2023-11-15T06:37:36.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.824+0000 7fd3148a6700 1 -- 172.21.15.123:0/3760689838 learned_addr learned my addr 172.21.15.123:0/3760689838 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:36.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.825+0000 7fd3148a6700 1 -- 172.21.15.123:0/3760689838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd30400b200 con 0x7fd3100703b0 2023-11-15T06:37:36.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.825+0000 7fd3148a6700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd31010cce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd30400b1d0 tx=0x7fd304006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:36.834 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.825+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd304011df0 con 0x7fd3100703b0 2023-11-15T06:37:36.835 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.825+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd30401b8c0 con 0x7fd3100703b0 2023-11-15T06:37:36.835 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.825+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd30401a830 con 0x7fd3100703b0 2023-11-15T06:37:36.835 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.825+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd31010d3d0 con 0x7fd3100703b0 2023-11-15T06:37:36.835 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.825+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd31010d7e0 con 0x7fd3100703b0 2023-11-15T06:37:36.836 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.826+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fd30400f070 con 0x7fd3100703b0 2023-11-15T06:37:36.837 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.826+0000 7fd30d7fa700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:36.837 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.826+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 --> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd310059100 con 0x7fd2f402ff00 2023-11-15T06:37:36.837 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.826+0000 7fd30ffff700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 msgr2=0x7fd2f40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1692115469 2023-11-15T06:37:36.837 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.826+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fd304040fb0 con 0x7fd3100703b0 2023-11-15T06:37:36.838 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:36.826+0000 7fd30ffff700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2023-11-15T06:37:37.029 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:37.027+0000 7fd30ffff700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 msgr2=0x7fd2f40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1692115469 2023-11-15T06:37:37.030 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:37.027+0000 7fd30ffff700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2023-11-15T06:37:37.430 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:37.428+0000 7fd30ffff700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 msgr2=0x7fd2f40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1692115469 2023-11-15T06:37:37.431 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:37.428+0000 7fd30ffff700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2023-11-15T06:37:38.231 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:38.229+0000 7fd30ffff700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 msgr2=0x7fd2f40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1692115469 2023-11-15T06:37:38.232 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:38.229+0000 7fd30ffff700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2023-11-15T06:37:39.832 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:39.830+0000 7fd30ffff700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 msgr2=0x7fd2f40323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.123:6800/1692115469 2023-11-15T06:37:39.833 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:39.830+0000 7fd30ffff700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2023-11-15T06:37:40.229 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:40.226+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fd304019d70 con 0x7fd3100703b0 2023-11-15T06:37:40.229 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:40.226+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 msgr2=0x7fd2f40323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2023-11-15T06:37:40.230 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:40.226+0000 7fd30d7fa700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:40.453 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:40.172612+0000 mon.smithi123 (mon.0) 91 : cluster [INF] Active manager daemon smithi123.razjtn restarted 2023-11-15T06:37:40.453 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:40.173059+0000 mon.smithi123 (mon.0) 92 : cluster [INF] Activating manager daemon smithi123.razjtn 2023-11-15T06:37:40.454 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:40.177288+0000 mon.smithi123 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2023-11-15T06:37:41.236 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.233+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fd304040c40 con 0x7fd3100703b0 2023-11-15T06:37:41.236 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.233+0000 7fd30d7fa700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd2f40339e0 0x7fd2f4035dd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:41.236 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.233+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 --> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fd310059100 con 0x7fd2f40339e0 2023-11-15T06:37:41.236 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.233+0000 7fd30ffff700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd2f40339e0 0x7fd2f4035dd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:41.237 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.234+0000 7fd30ffff700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd2f40339e0 0x7fd2f4035dd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd2fc006fd0 tx=0x7fd2fc006d70).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:41.237 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.234+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mgr.14162 v2:172.21.15.123:6800/1639427586 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fd310059100 con 0x7fd2f40339e0 2023-11-15T06:37:41.243 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.240+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 --> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fd310045ad0 con 0x7fd2f40339e0 2023-11-15T06:37:41.243 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd30d7fa700 1 -- 172.21.15.123:0/3760689838 <== mgr.14162 v2:172.21.15.123:6800/1639427586 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fd310045ad0 con 0x7fd2f40339e0 2023-11-15T06:37:41.244 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: { 2023-11-15T06:37:41.244 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2023-11-15T06:37:41.244 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: "initialized": true 2023-11-15T06:37:41.246 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: } 2023-11-15T06:37:41.246 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd2f40339e0 msgr2=0x7fd2f4035dd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:41.246 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd2f40339e0 0x7fd2f4035dd0 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fd2fc006fd0 tx=0x7fd2fc006d70).stop 2023-11-15T06:37:41.246 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 msgr2=0x7fd31010cce0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:41.246 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd31010cce0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fd30400b1d0 tx=0x7fd304006ee0).stop 2023-11-15T06:37:41.247 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 shutdown_connections 2023-11-15T06:37:41.247 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd3100703b0 0x7fd31010cce0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:41.247 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd2f40339e0 0x7fd2f4035dd0 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:41.247 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 --2- 172.21.15.123:0/3760689838 >> [v2:172.21.15.123:6800/1692115469,v1:172.21.15.123:6801/1692115469] conn(0x7fd2f402ff00 0x7fd2f40323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2023-11-15T06:37:41.247 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 >> 172.21.15.123:0/3760689838 conn(0x7fd3100f63c0 msgr2=0x7fd3100f7ea0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:41.248 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 shutdown_connections 2023-11-15T06:37:41.248 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:41.241+0000 7fd316b0a700 1 -- 172.21.15.123:0/3760689838 wait complete. 2023-11-15T06:37:41.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:40.228444+0000 mon.smithi123 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi123.razjtn(active, starting, since 0.0554718s) 2023-11-15T06:37:41.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.229765+0000 mon.smithi123 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata", "id": "smithi123"}]: dispatch 2023-11-15T06:37:41.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.230336+0000 mon.smithi123 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mgr metadata", "who": "smithi123.razjtn", "id": "smithi123.razjtn"}]: dispatch 2023-11-15T06:37:41.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.231385+0000 mon.smithi123 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mds metadata"}]: dispatch 2023-11-15T06:37:41.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.231523+0000 mon.smithi123 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "osd metadata"}]: dispatch 2023-11-15T06:37:41.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.231630+0000 mon.smithi123 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "mon metadata"}]: dispatch 2023-11-15T06:37:41.408 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:40.245212+0000 mon.smithi123 (mon.0) 100 : cluster [INF] Manager daemon smithi123.razjtn is now available 2023-11-15T06:37:41.408 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.343445+0000 mon.smithi123 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:41.408 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.345548+0000 mon.smithi123 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2023-11-15T06:37:41.408 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.360765+0000 mon.smithi123 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:37:41.408 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:40.363475+0000 mon.smithi123 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:37:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:mgr epoch 13 is available 2023-11-15T06:37:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:Generating a dashboard self-signed certificate... 2023-11-15T06:37:41.633 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2023-11-15T06:37:42.281 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.274+0000 7fa860f28700 1 Processor -- start 2023-11-15T06:37:42.281 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.274+0000 7fa860f28700 1 -- start start 2023-11-15T06:37:42.281 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa860f28700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c0fc0d0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:42.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa860f28700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c0fc5c0 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c0fc0d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:42.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c0fc0d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43506/0 (socket says 172.21.15.123:43506) 2023-11-15T06:37:42.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85a59c700 1 -- 172.21.15.123:0/2915723403 learned_addr learned my addr 172.21.15.123:0/2915723403 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:42.282 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85a59c700 1 -- 172.21.15.123:0/2915723403 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85c0fc700 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85a59c700 1 --2- 172.21.15.123:0/2915723403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c0fc0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa85000b5f0 tx=0x7fa850006ee0).ready entity=mon.0 client_cookie=62b1bd4dfb415007 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:42.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85959a700 1 -- 172.21.15.123:0/2915723403 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa85000d600 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85959a700 1 -- 172.21.15.123:0/2915723403 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa85000dc10 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.283 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.275+0000 7fa85959a700 1 -- 172.21.15.123:0/2915723403 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa850011cd0 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.284 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.276+0000 7fa860f28700 1 -- 172.21.15.123:0/2915723403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 msgr2=0x7fa85c0fc0d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:42.284 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.276+0000 7fa860f28700 1 --2- 172.21.15.123:0/2915723403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c0fc0d0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fa85000b5f0 tx=0x7fa850006ee0).stop 2023-11-15T06:37:42.284 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.276+0000 7fa860f28700 1 -- 172.21.15.123:0/2915723403 shutdown_connections 2023-11-15T06:37:42.284 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.276+0000 7fa860f28700 1 --2- 172.21.15.123:0/2915723403 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c0fc0d0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:42.284 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.276+0000 7fa860f28700 1 -- 172.21.15.123:0/2915723403 >> 172.21.15.123:0/2915723403 conn(0x7fa85c0f74b0 msgr2=0x7fa85c0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.276+0000 7fa860f28700 1 -- 172.21.15.123:0/2915723403 shutdown_connections 2023-11-15T06:37:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.276+0000 7fa860f28700 1 -- 172.21.15.123:0/2915723403 wait complete. 2023-11-15T06:37:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa860f28700 1 Processor -- start 2023-11-15T06:37:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa860f28700 1 -- start start 2023-11-15T06:37:42.285 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa860f28700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c10aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:42.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa860f28700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa85c10af40 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa85a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c10aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:42.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa85a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c10aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43510/0 (socket says 172.21.15.123:43510) 2023-11-15T06:37:42.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa85a59c700 1 -- 172.21.15.123:0/1599764276 learned_addr learned my addr 172.21.15.123:0/1599764276 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:42.286 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa85a59c700 1 -- 172.21.15.123:0/1599764276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa85000b200 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa85a59c700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c10aa50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa85000dfb0 tx=0x7fa850006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:42.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa84affd700 1 -- 172.21.15.123:0/1599764276 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa850011df0 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa84affd700 1 -- 172.21.15.123:0/1599764276 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa85001b8c0 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.287 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa85c10b140 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa84affd700 1 -- 172.21.15.123:0/1599764276 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa85001a8c0 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.277+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa85c10b550 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.278+0000 7fa84affd700 1 -- 172.21.15.123:0/1599764276 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fa85000f070 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.278+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa85c045ad0 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.278+0000 7fa84affd700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa84002ff00 0x7fa8400323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:42.288 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.279+0000 7fa84affd700 1 -- 172.21.15.123:0/1599764276 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa850041140 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.279+0000 7fa859d9b700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa84002ff00 0x7fa8400323c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.279+0000 7fa859d9b700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa84002ff00 0x7fa8400323c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:42.289 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.283+0000 7fa84affd700 1 -- 172.21.15.123:0/1599764276 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa850019d60 con 0x7fa85c0fbcb0 2023-11-15T06:37:42.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:42 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:41.234968+0000 mgr.smithi123.razjtn (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2023-11-15T06:37:42.433 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:42 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:41.235545+0000 mon.smithi123 (mon.0) 105 : cluster [DBG] mgrmap e15: smithi123.razjtn(active, since 1.06257s) 2023-11-15T06:37:42.433 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:42 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:41.240816+0000 mon.smithi123 (mon.0) 106 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:42.433 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:42 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:41.241617+0000 mgr.smithi123.razjtn (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2023-11-15T06:37:42.442 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.440+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 --> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fa85c10be30 con 0x7fa84002ff00 2023-11-15T06:37:42.517 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.514+0000 7fa84affd700 1 -- 172.21.15.123:0/1599764276 <== mgr.14162 v2:172.21.15.123:6800/1639427586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fa85c10be30 con 0x7fa84002ff00 2023-11-15T06:37:42.517 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: Self-signed certificate created 2023-11-15T06:37:42.519 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.515+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa84002ff00 msgr2=0x7fa8400323c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:42.519 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.515+0000 7fa860f28700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa84002ff00 0x7fa8400323c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fa84c0096a0 tx=0x7fa84c006b40).stop 2023-11-15T06:37:42.519 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.515+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 msgr2=0x7fa85c10aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:42.519 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.515+0000 7fa860f28700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c10aa50 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa85000dfb0 tx=0x7fa850006ee0).stop 2023-11-15T06:37:42.519 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.516+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 shutdown_connections 2023-11-15T06:37:42.520 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.516+0000 7fa860f28700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa84002ff00 0x7fa8400323c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:42.520 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.516+0000 7fa860f28700 1 --2- 172.21.15.123:0/1599764276 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa85c0fbcb0 0x7fa85c10aa50 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:42.520 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.516+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 >> 172.21.15.123:0/1599764276 conn(0x7fa85c0f74b0 msgr2=0x7fa85c0f8f30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:42.520 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.516+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 shutdown_connections 2023-11-15T06:37:42.520 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:42.516+0000 7fa860f28700 1 -- 172.21.15.123:0/1599764276 wait complete. 2023-11-15T06:37:42.816 INFO:teuthology.orchestra.run.smithi123.stderr:Creating initial admin user... 2023-11-15T06:37:42.816 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmp_cjvbug7:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2023-11-15T06:37:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.180+0000 7f8ab13cf700 1 Processor -- start 2023-11-15T06:37:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.180+0000 7f8ab13cf700 1 -- start start 2023-11-15T06:37:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.180+0000 7f8ab13cf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac070ae0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:43.186 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.180+0000 7f8ab13cf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac072960 con 0x7f8aac072540 2023-11-15T06:37:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aaaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac070ae0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aaaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac070ae0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43514/0 (socket says 172.21.15.123:43514) 2023-11-15T06:37:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aaaffd700 1 -- 172.21.15.123:0/461456354 learned_addr learned my addr 172.21.15.123:0/461456354 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aaaffd700 1 -- 172.21.15.123:0/461456354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aac070fd0 con 0x7f8aac072540 2023-11-15T06:37:43.187 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aaaffd700 1 --2- 172.21.15.123:0/461456354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac070ae0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f8aa000b5f0 tx=0x7f8aa0006ee0).ready entity=mon.0 client_cookie=1941252522d748e6 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:43.188 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aa9ffb700 1 -- 172.21.15.123:0/461456354 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa000d600 con 0x7f8aac072540 2023-11-15T06:37:43.188 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aa9ffb700 1 -- 172.21.15.123:0/461456354 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa000dc10 con 0x7f8aac072540 2023-11-15T06:37:43.188 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.181+0000 7f8aa9ffb700 1 -- 172.21.15.123:0/461456354 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa0011cd0 con 0x7f8aac072540 2023-11-15T06:37:43.188 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 -- 172.21.15.123:0/461456354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 msgr2=0x7f8aac070ae0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:43.188 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 --2- 172.21.15.123:0/461456354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac070ae0 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f8aa000b5f0 tx=0x7f8aa0006ee0).stop 2023-11-15T06:37:43.189 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 -- 172.21.15.123:0/461456354 shutdown_connections 2023-11-15T06:37:43.189 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 --2- 172.21.15.123:0/461456354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac070ae0 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:43.189 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 -- 172.21.15.123:0/461456354 >> 172.21.15.123:0/461456354 conn(0x7f8aac0f74b0 msgr2=0x7f8aac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:43.189 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 -- 172.21.15.123:0/461456354 shutdown_connections 2023-11-15T06:37:43.190 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 -- 172.21.15.123:0/461456354 wait complete. 2023-11-15T06:37:43.190 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.182+0000 7f8ab13cf700 1 Processor -- start 2023-11-15T06:37:43.190 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8ab13cf700 1 -- start start 2023-11-15T06:37:43.191 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8ab13cf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac10cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:43.191 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8ab13cf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8aac10d240 con 0x7f8aac072540 2023-11-15T06:37:43.191 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8aaaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac10cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:43.191 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8aaaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac10cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43520/0 (socket says 172.21.15.123:43520) 2023-11-15T06:37:43.191 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8aaaffd700 1 -- 172.21.15.123:0/1667090159 learned_addr learned my addr 172.21.15.123:0/1667090159 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:43.193 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8aaaffd700 1 -- 172.21.15.123:0/1667090159 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8aa000b200 con 0x7f8aac072540 2023-11-15T06:37:43.194 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8aaaffd700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac10cd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8aa000dfb0 tx=0x7f8aa0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:43.194 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8a937fe700 1 -- 172.21.15.123:0/1667090159 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa001b410 con 0x7f8aac072540 2023-11-15T06:37:43.194 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8a937fe700 1 -- 172.21.15.123:0/1667090159 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8aa001ba20 con 0x7f8aac072540 2023-11-15T06:37:43.194 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8aac10d440 con 0x7f8aac072540 2023-11-15T06:37:43.195 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8a937fe700 1 -- 172.21.15.123:0/1667090159 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8aa001a970 con 0x7f8aac072540 2023-11-15T06:37:43.195 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.183+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8aac10d850 con 0x7f8aac072540 2023-11-15T06:37:43.195 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.184+0000 7f8a937fe700 1 -- 172.21.15.123:0/1667090159 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f8aa000f070 con 0x7f8aac072540 2023-11-15T06:37:43.195 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.184+0000 7f8a937fe700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a9802feb0 0x7f8a98032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:43.195 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.184+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8aac045ad0 con 0x7f8aac072540 2023-11-15T06:37:43.195 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.184+0000 7f8a937fe700 1 -- 172.21.15.123:0/1667090159 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8aa00411e0 con 0x7f8aac072540 2023-11-15T06:37:43.196 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.185+0000 7f8aaa7fc700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a9802feb0 0x7f8a98032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:43.196 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.185+0000 7f8aaa7fc700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a9802feb0 0x7f8a98032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8a9c0096a0 tx=0x7f8a9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:43.196 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.189+0000 7f8a937fe700 1 -- 172.21.15.123:0/1667090159 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8aa0016070 con 0x7f8aac072540 2023-11-15T06:37:43.359 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.356+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 --> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f8aac106ce0 con 0x7f8a9802feb0 2023-11-15T06:37:43.577 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.575+0000 7f8a937fe700 1 -- 172.21.15.123:0/1667090159 <== mgr.14162 v2:172.21.15.123:6800/1639427586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f8aac106ce0 con 0x7f8a9802feb0 2023-11-15T06:37:43.578 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$5xu2JN.cn1Uzg3uJ3JWn0eA86T17T1L7vwYSxXgDVaBd0wDwTQlsG", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1700030263, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2023-11-15T06:37:43.580 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a9802feb0 msgr2=0x7f8a98032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:43.580 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a9802feb0 0x7f8a98032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f8a9c0096a0 tx=0x7f8a9c006b40).stop 2023-11-15T06:37:43.580 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 msgr2=0x7f8aac10cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:43.580 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac10cd50 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f8aa000dfb0 tx=0x7f8aa0006ee0).stop 2023-11-15T06:37:43.581 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 shutdown_connections 2023-11-15T06:37:43.581 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a9802feb0 0x7f8a98032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:43.581 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 --2- 172.21.15.123:0/1667090159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8aac072540 0x7f8aac10cd50 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:43.581 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.576+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 >> 172.21.15.123:0/1667090159 conn(0x7f8aac0f74b0 msgr2=0x7f8aac0f8f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:43.582 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.577+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 shutdown_connections 2023-11-15T06:37:43.582 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:43.577+0000 7f8ab13cf700 1 -- 172.21.15.123:0/1667090159 wait complete. 2023-11-15T06:37:43.606 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:42.243090+0000 mon.smithi123 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi123.razjtn(active, since 2s) 2023-11-15T06:37:43.607 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:42.441228+0000 mgr.smithi123.razjtn (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:43.607 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:42.511992+0000 mon.smithi123 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:43.607 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:42.514517+0000 mon.smithi123 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:43.933 INFO:teuthology.orchestra.run.smithi123.stderr:Fetching dashboard port number... 2023-11-15T06:37:43.933 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7:/var/log/ceph:z -v /tmp/ceph-tmpd9up4_f8:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmph_8r8dvu:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2023-11-15T06:37:44.316 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.311+0000 7fecef40c700 1 Processor -- start 2023-11-15T06:37:44.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.311+0000 7fecef40c700 1 -- start start 2023-11-15T06:37:44.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.311+0000 7fecef40c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece80fe9c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:44.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.311+0000 7fecef40c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece8070770 con 0x7fece8070350 2023-11-15T06:37:44.317 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7feced1a8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece80fe9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7feced1a8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece80fe9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43526/0 (socket says 172.21.15.123:43526) 2023-11-15T06:37:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7feced1a8700 1 -- 172.21.15.123:0/3930601867 learned_addr learned my addr 172.21.15.123:0/3930601867 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7feced1a8700 1 -- 172.21.15.123:0/3930601867 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fece80feeb0 con 0x7fece8070350 2023-11-15T06:37:44.318 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7feced1a8700 1 --2- 172.21.15.123:0/3930601867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece80fe9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fecd800b5f0 tx=0x7fecd8006ee0).ready entity=mon.0 client_cookie=65fa52af12ad246f server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7fecd7fff700 1 -- 172.21.15.123:0/3930601867 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd800d600 con 0x7fece8070350 2023-11-15T06:37:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7fecd7fff700 1 -- 172.21.15.123:0/3930601867 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecd800dc10 con 0x7fece8070350 2023-11-15T06:37:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.312+0000 7fecd7fff700 1 -- 172.21.15.123:0/3930601867 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd8011cd0 con 0x7fece8070350 2023-11-15T06:37:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 -- 172.21.15.123:0/3930601867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 msgr2=0x7fece80fe9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:44.319 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 --2- 172.21.15.123:0/3930601867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece80fe9c0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fecd800b5f0 tx=0x7fecd8006ee0).stop 2023-11-15T06:37:44.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 -- 172.21.15.123:0/3930601867 shutdown_connections 2023-11-15T06:37:44.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 --2- 172.21.15.123:0/3930601867 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece80fe9c0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:44.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 -- 172.21.15.123:0/3930601867 >> 172.21.15.123:0/3930601867 conn(0x7fece80f63c0 msgr2=0x7fece80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:44.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 -- 172.21.15.123:0/3930601867 shutdown_connections 2023-11-15T06:37:44.320 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 -- 172.21.15.123:0/3930601867 wait complete. 2023-11-15T06:37:44.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 Processor -- start 2023-11-15T06:37:44.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 -- start start 2023-11-15T06:37:44.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece810aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:44.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.313+0000 7fecef40c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fece810af40 con 0x7fece8070350 2023-11-15T06:37:44.321 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7feced1a8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece810aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:44.322 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7feced1a8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece810aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43530/0 (socket says 172.21.15.123:43530) 2023-11-15T06:37:44.322 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7feced1a8700 1 -- 172.21.15.123:0/1816120243 learned_addr learned my addr 172.21.15.123:0/1816120243 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:44.322 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7feced1a8700 1 -- 172.21.15.123:0/1816120243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fecd800b200 con 0x7fece8070350 2023-11-15T06:37:44.322 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7feced1a8700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece810aa50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fecd8006800 tx=0x7fecd8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:44.323 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7fecd5ffb700 1 -- 172.21.15.123:0/1816120243 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd8011df0 con 0x7fece8070350 2023-11-15T06:37:44.324 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7fecd5ffb700 1 -- 172.21.15.123:0/1816120243 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fecd801b8c0 con 0x7fece8070350 2023-11-15T06:37:44.324 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fece810b140 con 0x7fece8070350 2023-11-15T06:37:44.324 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7fecd5ffb700 1 -- 172.21.15.123:0/1816120243 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fecd801a890 con 0x7fece8070350 2023-11-15T06:37:44.324 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.314+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fece810b550 con 0x7fece8070350 2023-11-15T06:37:44.324 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.315+0000 7fecd5ffb700 1 -- 172.21.15.123:0/1816120243 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fecd800f070 con 0x7fece8070350 2023-11-15T06:37:44.325 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.315+0000 7fecd5ffb700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fecd002feb0 0x7fecd0032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:44.325 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.315+0000 7fecd5ffb700 1 -- 172.21.15.123:0/1816120243 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fecd8042070 con 0x7fece8070350 2023-11-15T06:37:44.325 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.315+0000 7fecec9a7700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fecd002feb0 0x7fecd0032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:44.325 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.315+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fece8059100 con 0x7fece8070350 2023-11-15T06:37:44.325 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.316+0000 7fecec9a7700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fecd002feb0 0x7fecd0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fece40096a0 tx=0x7fece4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:44.325 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.320+0000 7fecd5ffb700 1 -- 172.21.15.123:0/1816120243 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fecd8016070 con 0x7fece8070350 2023-11-15T06:37:44.472 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.470+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fece8045ad0 con 0x7fece8070350 2023-11-15T06:37:44.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.470+0000 7fecd5ffb700 1 -- 172.21.15.123:0/1816120243 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fecd8010bc0 con 0x7fece8070350 2023-11-15T06:37:44.473 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 8443 2023-11-15T06:37:44.475 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fecd002feb0 msgr2=0x7fecd0032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:44.475 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fecd002feb0 0x7fecd0032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fece40096a0 tx=0x7fece4006b40).stop 2023-11-15T06:37:44.475 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 msgr2=0x7fece810aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:44.476 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece810aa50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fecd8006800 tx=0x7fecd8006ee0).stop 2023-11-15T06:37:44.476 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 shutdown_connections 2023-11-15T06:37:44.476 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fecd002feb0 0x7fecd0032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:44.476 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 --2- 172.21.15.123:0/1816120243 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fece8070350 0x7fece810aa50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:44.476 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 >> 172.21.15.123:0/1816120243 conn(0x7fece80f63c0 msgr2=0x7fece80f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:44.477 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 shutdown_connections 2023-11-15T06:37:44.477 INFO:teuthology.orchestra.run.smithi123.stderr:/usr/bin/ceph: 2023-11-15T06:37:44.472+0000 7fecef40c700 1 -- 172.21.15.123:0/1816120243 wait complete. 2023-11-15T06:37:44.759 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:44 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:43.357926+0000 mgr.smithi123.razjtn (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:37:44.760 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:44 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:43.575629+0000 mon.smithi123 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:44.760 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:44 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:44.471003+0000 mon.smithi123 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.123:0/1816120243' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2023-11-15T06:37:44.841 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-enabled firewalld.service 2023-11-15T06:37:44.849 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: disabled 2023-11-15T06:37:44.849 INFO:teuthology.orchestra.run.smithi123.stderr:Running command: systemctl is-active firewalld.service 2023-11-15T06:37:44.856 INFO:teuthology.orchestra.run.smithi123.stderr:systemctl: inactive 2023-11-15T06:37:44.856 INFO:teuthology.orchestra.run.smithi123.stderr:firewalld.service is not enabled 2023-11-15T06:37:44.857 INFO:teuthology.orchestra.run.smithi123.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2023-11-15T06:37:44.859 INFO:teuthology.orchestra.run.smithi123.stderr:Ceph Dashboard is now available at: 2023-11-15T06:37:44.859 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.860 INFO:teuthology.orchestra.run.smithi123.stderr: URL: https://smithi123.front.sepia.ceph.com:8443/ 2023-11-15T06:37:44.860 INFO:teuthology.orchestra.run.smithi123.stderr: User: admin 2023-11-15T06:37:44.860 INFO:teuthology.orchestra.run.smithi123.stderr: Password: 8i1cpllajh 2023-11-15T06:37:44.860 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.860 INFO:teuthology.orchestra.run.smithi123.stderr:You can access the Ceph CLI with: 2023-11-15T06:37:44.861 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.861 INFO:teuthology.orchestra.run.smithi123.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2023-11-15T06:37:44.861 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.861 INFO:teuthology.orchestra.run.smithi123.stderr:Please consider enabling telemetry to help improve Ceph: 2023-11-15T06:37:44.861 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.862 INFO:teuthology.orchestra.run.smithi123.stderr: ceph telemetry on 2023-11-15T06:37:44.862 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.862 INFO:teuthology.orchestra.run.smithi123.stderr:For more information see: 2023-11-15T06:37:44.862 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.862 INFO:teuthology.orchestra.run.smithi123.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2023-11-15T06:37:44.863 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:37:44.863 INFO:teuthology.orchestra.run.smithi123.stderr:Bootstrap complete. 2023-11-15T06:37:44.863 INFO:teuthology.orchestra.run.smithi123.stderr:Releasing lock 139921009326568 on /run/cephadm/4a6dbac0-8381-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:37:44.863 INFO:teuthology.orchestra.run.smithi123.stderr:Lock 139921009326568 released on /run/cephadm/4a6dbac0-8381-11ee-8dbe-212e2dc638e7.lock 2023-11-15T06:37:44.892 INFO:tasks.cephadm:Fetching config... 2023-11-15T06:37:44.893 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:37:44.893 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2023-11-15T06:37:44.911 INFO:tasks.cephadm:Fetching client.admin keyring... 2023-11-15T06:37:44.911 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:37:44.911 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2023-11-15T06:37:44.970 INFO:tasks.cephadm:Fetching mon keyring... 2023-11-15T06:37:44.971 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:37:44.972 DEBUG:teuthology.orchestra.run.smithi123:> sudo dd if=/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/mon.smithi123/keyring of=/dev/stdout 2023-11-15T06:37:45.041 INFO:tasks.cephadm:Fetching pub ssh key... 2023-11-15T06:37:45.042 DEBUG:teuthology.orchestra.run.smithi123:> set -ex 2023-11-15T06:37:45.042 DEBUG:teuthology.orchestra.run.smithi123:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2023-11-15T06:37:45.100 INFO:tasks.cephadm:Installing pub ssh key for root users... 2023-11-15T06:37:45.100 DEBUG:teuthology.orchestra.run.smithi123:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbbBRJvpr3wJrixHcdvgQOVMeGSYrjFeIM8VMEQTQ+WUkT9kF5yep0whX/uD/sr6Zae8XhvejMRhbvovoHqHMCUx/TADUSeFm4R9ntKnrtAjybfy519BKJV7IjYA+a1mPA7Q9x+TZlxbVRFmixugoHMZF5yoWXiFX7qWTxsPvgUTF7I1Fl37B+zGVodA9IAZqOwtSeEqUKwLhI3FvI18ZDOQ9IEKkmaHjS2dRqQd3flORGF2dX/dr//ezJifMRADAMkpyZDGE2NdlXrrM358or09gtRtPvHqqczy4qrSzV128tDCxvGvgab7TEdYFAgUKL+YaAW/iKNeaG2pixhhsVtCXS+2Tg0h9rGZ2uIfs/Ab3jTj3e3fLRZ8sqtQ0nSfZRo8ySWdc61aDvhGUKTd+tdPg/N7AJLgZDgZGIzgIhQdT9mp7fclKup/fs11zURCQSeTROpKO3drZ/c5bDG+TBDt4YvObR5gSfiF6XtRbkv+rBf6o2BXwNy+hE29RXlU0= ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T06:37:45.178 INFO:teuthology.orchestra.run.smithi123.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbbBRJvpr3wJrixHcdvgQOVMeGSYrjFeIM8VMEQTQ+WUkT9kF5yep0whX/uD/sr6Zae8XhvejMRhbvovoHqHMCUx/TADUSeFm4R9ntKnrtAjybfy519BKJV7IjYA+a1mPA7Q9x+TZlxbVRFmixugoHMZF5yoWXiFX7qWTxsPvgUTF7I1Fl37B+zGVodA9IAZqOwtSeEqUKwLhI3FvI18ZDOQ9IEKkmaHjS2dRqQd3flORGF2dX/dr//ezJifMRADAMkpyZDGE2NdlXrrM358or09gtRtPvHqqczy4qrSzV128tDCxvGvgab7TEdYFAgUKL+YaAW/iKNeaG2pixhhsVtCXS+2Tg0h9rGZ2uIfs/Ab3jTj3e3fLRZ8sqtQ0nSfZRo8ySWdc61aDvhGUKTd+tdPg/N7AJLgZDgZGIzgIhQdT9mp7fclKup/fs11zURCQSeTROpKO3drZ/c5bDG+TBDt4YvObR5gSfiF6XtRbkv+rBf6o2BXwNy+hE29RXlU0= ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:37:45.190 DEBUG:teuthology.orchestra.run.smithi135:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbbBRJvpr3wJrixHcdvgQOVMeGSYrjFeIM8VMEQTQ+WUkT9kF5yep0whX/uD/sr6Zae8XhvejMRhbvovoHqHMCUx/TADUSeFm4R9ntKnrtAjybfy519BKJV7IjYA+a1mPA7Q9x+TZlxbVRFmixugoHMZF5yoWXiFX7qWTxsPvgUTF7I1Fl37B+zGVodA9IAZqOwtSeEqUKwLhI3FvI18ZDOQ9IEKkmaHjS2dRqQd3flORGF2dX/dr//ezJifMRADAMkpyZDGE2NdlXrrM358or09gtRtPvHqqczy4qrSzV128tDCxvGvgab7TEdYFAgUKL+YaAW/iKNeaG2pixhhsVtCXS+2Tg0h9rGZ2uIfs/Ab3jTj3e3fLRZ8sqtQ0nSfZRo8ySWdc61aDvhGUKTd+tdPg/N7AJLgZDgZGIzgIhQdT9mp7fclKup/fs11zURCQSeTROpKO3drZ/c5bDG+TBDt4YvObR5gSfiF6XtRbkv+rBf6o2BXwNy+hE29RXlU0= ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2023-11-15T06:37:45.237 INFO:teuthology.orchestra.run.smithi135.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDbbBRJvpr3wJrixHcdvgQOVMeGSYrjFeIM8VMEQTQ+WUkT9kF5yep0whX/uD/sr6Zae8XhvejMRhbvovoHqHMCUx/TADUSeFm4R9ntKnrtAjybfy519BKJV7IjYA+a1mPA7Q9x+TZlxbVRFmixugoHMZF5yoWXiFX7qWTxsPvgUTF7I1Fl37B+zGVodA9IAZqOwtSeEqUKwLhI3FvI18ZDOQ9IEKkmaHjS2dRqQd3flORGF2dX/dr//ezJifMRADAMkpyZDGE2NdlXrrM358or09gtRtPvHqqczy4qrSzV128tDCxvGvgab7TEdYFAgUKL+YaAW/iKNeaG2pixhhsVtCXS+2Tg0h9rGZ2uIfs/Ab3jTj3e3fLRZ8sqtQ0nSfZRo8ySWdc61aDvhGUKTd+tdPg/N7AJLgZDgZGIzgIhQdT9mp7fclKup/fs11zURCQSeTROpKO3drZ/c5bDG+TBDt4YvObR5gSfiF6XtRbkv+rBf6o2BXwNy+hE29RXlU0= ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7 2023-11-15T06:37:45.249 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2023-11-15T06:37:47.023 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 -- 172.21.15.123:0/2026802159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 msgr2=0x7f0ab80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:47.024 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 --2- 172.21.15.123:0/2026802159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab80f90f0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f0aac009fb0 tx=0x7f0aac009b30).stop 2023-11-15T06:37:47.024 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 -- 172.21.15.123:0/2026802159 shutdown_connections 2023-11-15T06:37:47.024 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 --2- 172.21.15.123:0/2026802159 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab80f90f0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:47.025 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 -- 172.21.15.123:0/2026802159 >> 172.21.15.123:0/2026802159 conn(0x7f0ab80f0b70 msgr2=0x7f0ab80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:47.025 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 -- 172.21.15.123:0/2026802159 shutdown_connections 2023-11-15T06:37:47.025 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 -- 172.21.15.123:0/2026802159 wait complete. 2023-11-15T06:37:47.025 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 Processor -- start 2023-11-15T06:37:47.025 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.019+0000 7f0abef11700 1 -- start start 2023-11-15T06:37:47.026 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abef11700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab81066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:47.026 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abef11700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0ab8106ba0 con 0x7f0ab8060280 2023-11-15T06:37:47.026 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abccad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab81066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:47.026 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abccad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab81066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:43554/0 (socket says 172.21.15.123:43554) 2023-11-15T06:37:47.026 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abccad700 1 -- 172.21.15.123:0/1762505858 learned_addr learned my addr 172.21.15.123:0/1762505858 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:37:47.027 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abccad700 1 -- 172.21.15.123:0/1762505858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0aac005040 con 0x7f0ab8060280 2023-11-15T06:37:47.027 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abccad700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab81066b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0aac013510 tx=0x7f0aac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:47.027 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0ab57fa700 1 -- 172.21.15.123:0/1762505858 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac009320 con 0x7f0ab8060280 2023-11-15T06:37:47.027 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0ab57fa700 1 -- 172.21.15.123:0/1762505858 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0aac013f00 con 0x7f0ab8060280 2023-11-15T06:37:47.027 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0ab8106da0 con 0x7f0ab8060280 2023-11-15T06:37:47.028 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0ab57fa700 1 -- 172.21.15.123:0/1762505858 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0aac014dc0 con 0x7f0ab8060280 2023-11-15T06:37:47.028 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.020+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0ab8107230 con 0x7f0ab8060280 2023-11-15T06:37:47.028 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.021+0000 7f0ab57fa700 1 -- 172.21.15.123:0/1762505858 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0aac0263a0 con 0x7f0ab8060280 2023-11-15T06:37:47.028 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.021+0000 7f0ab57fa700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0aa802ffb0 0x7f0aa8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:37:47.028 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.021+0000 7f0ab57fa700 1 -- 172.21.15.123:0/1762505858 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0aac040fe0 con 0x7f0ab8060280 2023-11-15T06:37:47.029 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.021+0000 7f0ab7fff700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0aa802ffb0 0x7f0aa8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:37:47.029 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.021+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0ab8059100 con 0x7f0ab8060280 2023-11-15T06:37:47.029 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.022+0000 7f0ab7fff700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0aa802ffb0 0x7f0aa8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0aa40096a0 tx=0x7f0aa4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:37:47.030 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.025+0000 7f0ab57fa700 1 -- 172.21.15.123:0/1762505858 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0aac012920 con 0x7f0ab8060280 2023-11-15T06:37:47.188 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.185+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7f0ab8045ad0 con 0x7f0ab8060280 2023-11-15T06:37:47.192 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.190+0000 7f0ab57fa700 1 -- 172.21.15.123:0/1762505858 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7f0aac02b020 con 0x7f0ab8060280 2023-11-15T06:37:47.194 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.191+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0aa802ffb0 msgr2=0x7f0aa8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:47.194 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.191+0000 7f0abef11700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0aa802ffb0 0x7f0aa8032470 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f0aa40096a0 tx=0x7f0aa4006b40).stop 2023-11-15T06:37:47.194 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.191+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 msgr2=0x7f0ab81066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:37:47.195 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.191+0000 7f0abef11700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab81066b0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f0aac013510 tx=0x7f0aac00b040).stop 2023-11-15T06:37:47.195 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.192+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 shutdown_connections 2023-11-15T06:37:47.195 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.192+0000 7f0abef11700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0aa802ffb0 0x7f0aa8032470 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:47.195 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.192+0000 7f0abef11700 1 --2- 172.21.15.123:0/1762505858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0ab8060280 0x7f0ab81066b0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:37:47.195 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.192+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 >> 172.21.15.123:0/1762505858 conn(0x7f0ab80f0b70 msgr2=0x7f0ab80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:37:47.196 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.192+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 shutdown_connections 2023-11-15T06:37:47.196 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:37:47.192+0000 7f0abef11700 1 -- 172.21.15.123:0/1762505858 wait complete. 2023-11-15T06:37:47.519 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:37:46.251516+0000 mon.smithi123 (mon.0) 112 : cluster [DBG] mgrmap e17: smithi123.razjtn(active, since 6s) 2023-11-15T06:37:47.519 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:47.190612+0000 mon.smithi123 (mon.0) 113 : audit [INF] from='client.? 172.21.15.123:0/1762505858' entity='client.admin' 2023-11-15T06:37:47.587 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi135 2023-11-15T06:37:47.588 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:37:47.588 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.conf 2023-11-15T06:37:47.610 DEBUG:teuthology.orchestra.run.smithi135:> set -ex 2023-11-15T06:37:47.611 DEBUG:teuthology.orchestra.run.smithi135:> dd of=/etc/ceph/ceph.client.admin.keyring 2023-11-15T06:37:47.671 INFO:tasks.cephadm:Adding host smithi135 to orchestrator... 2023-11-15T06:37:47.671 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph orch host add smithi135 2023-11-15T06:37:49.657 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:48.373481+0000 mon.smithi123 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:49.658 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:37:48.624573+0000 mon.smithi123 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:37:49.658 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:37:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:37:48.630813+0000 mgr.smithi123.razjtn (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi123 on smithi123 2023-11-15T06:38:01.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:00.232914+0000 mgr.smithi123.razjtn (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:02.233296+0000 mgr.smithi123.razjtn (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:05.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:04.233589+0000 mgr.smithi123.razjtn (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:07.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:06.233996+0000 mgr.smithi123.razjtn (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:08.234484+0000 mgr.smithi123.razjtn (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:10.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.265+0000 7fc85d0fa700 1 -- 172.21.15.135:0/1814859439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 msgr2=0x7fc8580f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:10.268 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.265+0000 7fc85d0fa700 1 --2- 172.21.15.135:0/1814859439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc8580f5830 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fc84c009fb0 tx=0x7fc84c009b30).stop 2023-11-15T06:38:10.269 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.265+0000 7fc85d0fa700 1 -- 172.21.15.135:0/1814859439 shutdown_connections 2023-11-15T06:38:10.269 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.265+0000 7fc85d0fa700 1 --2- 172.21.15.135:0/1814859439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc8580f5830 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:10.269 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.265+0000 7fc85d0fa700 1 -- 172.21.15.135:0/1814859439 >> 172.21.15.135:0/1814859439 conn(0x7fc8580f0b70 msgr2=0x7fc8580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:10.269 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.265+0000 7fc85d0fa700 1 -- 172.21.15.135:0/1814859439 shutdown_connections 2023-11-15T06:38:10.269 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.265+0000 7fc85d0fa700 1 -- 172.21.15.135:0/1814859439 wait complete. 2023-11-15T06:38:10.270 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.266+0000 7fc85d0fa700 1 Processor -- start 2023-11-15T06:38:10.270 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.266+0000 7fc85d0fa700 1 -- start start 2023-11-15T06:38:10.272 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.266+0000 7fc85d0fa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc858109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:10.272 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.266+0000 7fc85d0fa700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc858060220 con 0x7fc8580f5410 2023-11-15T06:38:10.272 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.266+0000 7fc856d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc858109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:10.272 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.266+0000 7fc856d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc858109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:60614/0 (socket says 172.21.15.135:60614) 2023-11-15T06:38:10.272 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.266+0000 7fc856d9d700 1 -- 172.21.15.135:0/3887401577 learned_addr learned my addr 172.21.15.135:0/3887401577 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:10.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.267+0000 7fc856d9d700 1 -- 172.21.15.135:0/3887401577 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc84c005040 con 0x7fc8580f5410 2023-11-15T06:38:10.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.267+0000 7fc856d9d700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc858109450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc84c013510 tx=0x7fc84c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:10.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.267+0000 7fc83f7fe700 1 -- 172.21.15.135:0/3887401577 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc84c009320 con 0x7fc8580f5410 2023-11-15T06:38:10.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.267+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc858060420 con 0x7fc8580f5410 2023-11-15T06:38:10.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.267+0000 7fc83f7fe700 1 -- 172.21.15.135:0/3887401577 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc84c013f00 con 0x7fc8580f5410 2023-11-15T06:38:10.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.267+0000 7fc83f7fe700 1 -- 172.21.15.135:0/3887401577 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc84c014d90 con 0x7fc8580f5410 2023-11-15T06:38:10.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.267+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc858109b00 con 0x7fc8580f5410 2023-11-15T06:38:10.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.268+0000 7fc83f7fe700 1 -- 172.21.15.135:0/3887401577 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc84c014420 con 0x7fc8580f5410 2023-11-15T06:38:10.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.268+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc858059100 con 0x7fc8580f5410 2023-11-15T06:38:10.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.268+0000 7fc83f7fe700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc84002ff60 0x7fc840032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:10.275 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.268+0000 7fc83f7fe700 1 -- 172.21.15.135:0/3887401577 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc84c041000 con 0x7fc8580f5410 2023-11-15T06:38:10.275 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.268+0000 7fc85659c700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc84002ff60 0x7fc840032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:10.275 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.269+0000 7fc85659c700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc84002ff60 0x7fc840032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc8480096a0 tx=0x7fc848006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:10.275 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.273+0000 7fc83f7fe700 1 -- 172.21.15.135:0/3887401577 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc84c0095f0 con 0x7fc8580f5410 2023-11-15T06:38:10.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:10.444+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 --> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}) v1 -- 0x7fc8580f1d70 con 0x7fc84002ff60 2023-11-15T06:38:11.042 INFO:teuthology.orchestra.run.smithi135.stdout:Added host 'smithi135' 2023-11-15T06:38:11.043 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.039+0000 7fc83f7fe700 1 -- 172.21.15.135:0/3887401577 <== mgr.14162 v2:172.21.15.123:6800/1639427586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fc8580f1d70 con 0x7fc84002ff60 2023-11-15T06:38:11.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc84002ff60 msgr2=0x7fc840032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:11.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc84002ff60 0x7fc840032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fc8480096a0 tx=0x7fc848006b40).stop 2023-11-15T06:38:11.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 msgr2=0x7fc858109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:11.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc858109450 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7fc84c013510 tx=0x7fc84c00b040).stop 2023-11-15T06:38:11.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 shutdown_connections 2023-11-15T06:38:11.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc84002ff60 0x7fc840032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:11.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 --2- 172.21.15.135:0/3887401577 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8580f5410 0x7fc858109450 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:11.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 >> 172.21.15.135:0/3887401577 conn(0x7fc8580f0b70 msgr2=0x7fc8580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:11.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 shutdown_connections 2023-11-15T06:38:11.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:11.041+0000 7fc85d0fa700 1 -- 172.21.15.135:0/3887401577 wait complete. 2023-11-15T06:38:11.373 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph orch host ls --format=json 2023-11-15T06:38:12.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:12 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:10.234733+0000 mgr.smithi123.razjtn (mgr.14162) 11 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:12.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:12 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:10.445963+0000 mgr.smithi123.razjtn (mgr.14162) 12 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi135", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:38:12.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:12 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:11.039174+0000 mon.smithi123 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:38:12.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:12 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:38:11.039617+0000 mgr.smithi123.razjtn (mgr.14162) 13 : cephadm [INF] Added host smithi135 2023-11-15T06:38:12.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.478+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1814561334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 msgr2=0x7f4eb00f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:12.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.478+0000 7f4eb4c6c700 1 --2- 172.21.15.135:0/1814561334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb00f7d00 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f4ea0009fb0 tx=0x7f4ea0009b30).stop 2023-11-15T06:38:12.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.478+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1814561334 shutdown_connections 2023-11-15T06:38:12.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.478+0000 7f4eb4c6c700 1 --2- 172.21.15.135:0/1814561334 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb00f7d00 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:12.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.478+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1814561334 >> 172.21.15.135:0/1814561334 conn(0x7f4eb00f0120 msgr2=0x7f4eb00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:12.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.478+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1814561334 shutdown_connections 2023-11-15T06:38:12.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.478+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1814561334 wait complete. 2023-11-15T06:38:12.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.479+0000 7f4eb4c6c700 1 Processor -- start 2023-11-15T06:38:12.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.479+0000 7f4eb4c6c700 1 -- start start 2023-11-15T06:38:12.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.479+0000 7f4eb4c6c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb01064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:12.485 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.479+0000 7f4eb4c6c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4eb01069d0 con 0x7f4eb00f78e0 2023-11-15T06:38:12.485 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.479+0000 7f4eaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb01064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:12.485 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4eaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb01064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:60638/0 (socket says 172.21.15.135:60638) 2023-11-15T06:38:12.485 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4eaf7fe700 1 -- 172.21.15.135:0/1806559445 learned_addr learned my addr 172.21.15.135:0/1806559445 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:12.485 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4eaf7fe700 1 -- 172.21.15.135:0/1806559445 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ea0005040 con 0x7f4eb00f78e0 2023-11-15T06:38:12.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4eaf7fe700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb01064e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f4ea0013510 tx=0x7f4ea000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:12.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4e97fff700 1 -- 172.21.15.135:0/1806559445 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea0009320 con 0x7f4eb00f78e0 2023-11-15T06:38:12.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4eb0106bd0 con 0x7f4eb00f78e0 2023-11-15T06:38:12.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4e97fff700 1 -- 172.21.15.135:0/1806559445 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ea0013f00 con 0x7f4eb00f78e0 2023-11-15T06:38:12.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4e97fff700 1 -- 172.21.15.135:0/1806559445 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ea0014dc0 con 0x7f4eb00f78e0 2023-11-15T06:38:12.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.480+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4eb0107060 con 0x7f4eb00f78e0 2023-11-15T06:38:12.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.481+0000 7f4e97fff700 1 -- 172.21.15.135:0/1806559445 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ea0014420 con 0x7f4eb00f78e0 2023-11-15T06:38:12.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.481+0000 7f4e97fff700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4e9802ff60 0x7f4e98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:12.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.481+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4eb0045ad0 con 0x7f4eb00f78e0 2023-11-15T06:38:12.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.481+0000 7f4e97fff700 1 -- 172.21.15.135:0/1806559445 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ea0041110 con 0x7f4eb00f78e0 2023-11-15T06:38:12.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.481+0000 7f4eaeffd700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4e9802ff60 0x7f4e98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:12.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.482+0000 7f4eaeffd700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4e9802ff60 0x7f4e98032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4ea40096a0 tx=0x7f4ea4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:12.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.485+0000 7f4e97fff700 1 -- 172.21.15.135:0/1806559445 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ea001cb10 con 0x7f4eb00f78e0 2023-11-15T06:38:12.660 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.657+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 --> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4eb00f12d0 con 0x7f4e9802ff60 2023-11-15T06:38:12.661 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.659+0000 7f4e97fff700 1 -- 172.21.15.135:0/1806559445 <== mgr.14162 v2:172.21.15.123:6800/1639427586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f4eb00f12d0 con 0x7f4e9802ff60 2023-11-15T06:38:12.662 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:12.662 INFO:teuthology.orchestra.run.smithi135.stdout:[{"addr": "172.21.15.123", "hostname": "smithi123", "labels": [], "status": ""}, {"addr": "smithi135", "hostname": "smithi135", "labels": [], "status": ""}] 2023-11-15T06:38:12.663 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4e9802ff60 msgr2=0x7f4e98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:12.663 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4e9802ff60 0x7f4e98032420 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4ea40096a0 tx=0x7f4ea4006b40).stop 2023-11-15T06:38:12.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 msgr2=0x7f4eb01064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:12.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb01064e0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f4ea0013510 tx=0x7f4ea000b040).stop 2023-11-15T06:38:12.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 shutdown_connections 2023-11-15T06:38:12.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4e9802ff60 0x7f4e98032420 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:12.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 --2- 172.21.15.135:0/1806559445 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4eb00f78e0 0x7f4eb01064e0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:12.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 >> 172.21.15.135:0/1806559445 conn(0x7f4eb00f0120 msgr2=0x7f4eb00f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:12.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 shutdown_connections 2023-11-15T06:38:12.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:12.661+0000 7f4eb4c6c700 1 -- 172.21.15.135:0/1806559445 wait complete. 2023-11-15T06:38:13.030 INFO:tasks.cephadm:Setting crush tunables to default 2023-11-15T06:38:13.031 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph osd crush tunables default 2023-11-15T06:38:14.180 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.174+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2469455039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 msgr2=0x7fe9ec0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.174+0000 7fe9f30e4700 1 --2- 172.21.15.123:0/2469455039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec0f90f0 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7fe9e0009fb0 tx=0x7fe9e0009b30).stop 2023-11-15T06:38:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.174+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2469455039 shutdown_connections 2023-11-15T06:38:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.174+0000 7fe9f30e4700 1 --2- 172.21.15.123:0/2469455039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec0f90f0 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:14.181 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.174+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2469455039 >> 172.21.15.123:0/2469455039 conn(0x7fe9ec0f0b70 msgr2=0x7fe9ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.174+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2469455039 shutdown_connections 2023-11-15T06:38:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.174+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2469455039 wait complete. 2023-11-15T06:38:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f30e4700 1 Processor -- start 2023-11-15T06:38:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f30e4700 1 -- start start 2023-11-15T06:38:14.182 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f30e4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f30e4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe9ec106ba0 con 0x7fe9ec060280 2023-11-15T06:38:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f0e80700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f0e80700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.123:51836/0 (socket says 172.21.15.123:51836) 2023-11-15T06:38:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f0e80700 1 -- 172.21.15.123:0/2363318842 learned_addr learned my addr 172.21.15.123:0/2363318842 (peer_addr_for_me v2:172.21.15.123:0/0) 2023-11-15T06:38:14.183 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f0e80700 1 -- 172.21.15.123:0/2363318842 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe9e0005040 con 0x7fe9ec060280 2023-11-15T06:38:14.186 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f0e80700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec1066b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe9e0013510 tx=0x7fe9e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:14.186 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9e97fa700 1 -- 172.21.15.123:0/2363318842 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9e0009320 con 0x7fe9ec060280 2023-11-15T06:38:14.186 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9e97fa700 1 -- 172.21.15.123:0/2363318842 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9e0013f00 con 0x7fe9ec060280 2023-11-15T06:38:14.187 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe9ec106da0 con 0x7fe9ec060280 2023-11-15T06:38:14.187 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.175+0000 7fe9e97fa700 1 -- 172.21.15.123:0/2363318842 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9e0014dc0 con 0x7fe9ec060280 2023-11-15T06:38:14.187 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.176+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9ec107230 con 0x7fe9ec060280 2023-11-15T06:38:14.188 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.176+0000 7fe9e97fa700 1 -- 172.21.15.123:0/2363318842 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9e00263a0 con 0x7fe9ec060280 2023-11-15T06:38:14.188 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.176+0000 7fe9e97fa700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe9dc02ffb0 0x7fe9dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:14.188 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.176+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe9ec045ad0 con 0x7fe9ec060280 2023-11-15T06:38:14.188 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.176+0000 7fe9e97fa700 1 -- 172.21.15.123:0/2363318842 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9e0040fc0 con 0x7fe9ec060280 2023-11-15T06:38:14.189 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.176+0000 7fe9ebfff700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe9dc02ffb0 0x7fe9dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:14.189 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.177+0000 7fe9ebfff700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe9dc02ffb0 0x7fe9dc032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe9d80096a0 tx=0x7fe9d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:14.189 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.182+0000 7fe9e97fa700 1 -- 172.21.15.123:0/2363318842 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe9e0012920 con 0x7fe9ec060280 2023-11-15T06:38:14.340 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:14.337+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7fe9ec1080e0 con 0x7fe9ec060280 2023-11-15T06:38:14.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:14 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:12.235038+0000 mgr.smithi123.razjtn (mgr.14162) 14 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:14.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:14 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:12.659776+0000 mgr.smithi123.razjtn (mgr.14162) 15 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2023-11-15T06:38:15.046 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.043+0000 7fe9e97fa700 1 -- 172.21.15.123:0/2363318842 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7fe9e0012330 con 0x7fe9ec060280 2023-11-15T06:38:15.047 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe9dc02ffb0 msgr2=0x7fe9dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe9dc02ffb0 0x7fe9dc032470 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fe9d80096a0 tx=0x7fe9d8006b40).stop 2023-11-15T06:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 msgr2=0x7fe9ec1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec1066b0 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fe9e0013510 tx=0x7fe9e000b040).stop 2023-11-15T06:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 shutdown_connections 2023-11-15T06:38:15.048 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe9dc02ffb0 0x7fe9dc032470 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 --2- 172.21.15.123:0/2363318842 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9ec060280 0x7fe9ec1066b0 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 >> 172.21.15.123:0/2363318842 conn(0x7fe9ec0f0b70 msgr2=0x7fe9ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 shutdown_connections 2023-11-15T06:38:15.049 INFO:teuthology.orchestra.run.smithi123.stderr:2023-11-15T06:38:15.045+0000 7fe9f30e4700 1 -- 172.21.15.123:0/2363318842 wait complete. 2023-11-15T06:38:15.050 INFO:teuthology.orchestra.run.smithi123.stderr:adjusted tunables profile to default 2023-11-15T06:38:15.346 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:14.339213+0000 mon.smithi123 (mon.0) 117 : audit [INF] from='client.? 172.21.15.123:0/2363318842' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2023-11-15T06:38:15.447 INFO:tasks.cephadm:Adding mon.smithi123 on smithi123 2023-11-15T06:38:15.447 INFO:tasks.cephadm:Adding mon.smithi135 on smithi135 2023-11-15T06:38:15.447 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph orch apply mon '2;smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135' 2023-11-15T06:38:16.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:16 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:14.235276+0000 mgr.smithi123.razjtn (mgr.14162) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:16.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:16 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:15.044338+0000 mon.smithi123 (mon.0) 118 : audit [INF] from='client.? 172.21.15.123:0/2363318842' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2023-11-15T06:38:16.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:16 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:15.044381+0000 mon.smithi123 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2023-11-15T06:38:16.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.574+0000 7f2615a36700 1 -- 172.21.15.135:0/2685921964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 msgr2=0x7f26100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:16.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.574+0000 7f2615a36700 1 --2- 172.21.15.135:0/2685921964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f26100f5830 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f2604009fb0 tx=0x7f2604009b30).stop 2023-11-15T06:38:16.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.574+0000 7f2615a36700 1 -- 172.21.15.135:0/2685921964 shutdown_connections 2023-11-15T06:38:16.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.574+0000 7f2615a36700 1 --2- 172.21.15.135:0/2685921964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f26100f5830 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:16.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.574+0000 7f2615a36700 1 -- 172.21.15.135:0/2685921964 >> 172.21.15.135:0/2685921964 conn(0x7f26100f0b70 msgr2=0x7f26100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:16.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.574+0000 7f2615a36700 1 -- 172.21.15.135:0/2685921964 shutdown_connections 2023-11-15T06:38:16.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.574+0000 7f2615a36700 1 -- 172.21.15.135:0/2685921964 wait complete. 2023-11-15T06:38:16.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.575+0000 7f2615a36700 1 Processor -- start 2023-11-15T06:38:16.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.575+0000 7f2615a36700 1 -- start start 2023-11-15T06:38:16.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.575+0000 7f2615a36700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f2610106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:16.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.575+0000 7f2615a36700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2610106c10 con 0x7f26100f5410 2023-11-15T06:38:16.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.575+0000 7f260effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f2610106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:16.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.575+0000 7f260effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f2610106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:60662/0 (socket says 172.21.15.135:60662) 2023-11-15T06:38:16.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.575+0000 7f260effd700 1 -- 172.21.15.135:0/3146717475 learned_addr learned my addr 172.21.15.135:0/3146717475 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:16.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.576+0000 7f260effd700 1 -- 172.21.15.135:0/3146717475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2604005040 con 0x7f26100f5410 2023-11-15T06:38:16.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.576+0000 7f260effd700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f2610106720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f2604013510 tx=0x7f260400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:16.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.576+0000 7f25f77fe700 1 -- 172.21.15.135:0/3146717475 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2604009320 con 0x7f26100f5410 2023-11-15T06:38:16.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.576+0000 7f25f77fe700 1 -- 172.21.15.135:0/3146717475 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2604013f00 con 0x7f26100f5410 2023-11-15T06:38:16.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.576+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2610106e10 con 0x7f26100f5410 2023-11-15T06:38:16.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.576+0000 7f25f77fe700 1 -- 172.21.15.135:0/3146717475 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2604014d90 con 0x7f26100f5410 2023-11-15T06:38:16.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.576+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26101072a0 con 0x7f26100f5410 2023-11-15T06:38:16.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.577+0000 7f25f77fe700 1 -- 172.21.15.135:0/3146717475 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f26040263a0 con 0x7f26100f5410 2023-11-15T06:38:16.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.577+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2610045ad0 con 0x7f26100f5410 2023-11-15T06:38:16.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.577+0000 7f25f77fe700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f802ff60 0x7f25f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:16.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.577+0000 7f25f77fe700 1 -- 172.21.15.135:0/3146717475 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2604041310 con 0x7f26100f5410 2023-11-15T06:38:16.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.577+0000 7f260e7fc700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f802ff60 0x7f25f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:16.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.578+0000 7f260e7fc700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f802ff60 0x7f25f8032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f26000096a0 tx=0x7f2600006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:16.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.582+0000 7f25f77fe700 1 -- 172.21.15.135:0/3146717475 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f260401d3d0 con 0x7f26100f5410 2023-11-15T06:38:16.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.758+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 --> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135", "target": ["mon-mgr", ""]}) v1 -- 0x7f26100f2a80 con 0x7f25f802ff60 2023-11-15T06:38:16.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.764+0000 7f25f77fe700 1 -- 172.21.15.135:0/3146717475 <== mgr.14162 v2:172.21.15.123:6800/1639427586 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f26100f2a80 con 0x7f25f802ff60 2023-11-15T06:38:16.767 INFO:teuthology.orchestra.run.smithi135.stdout:Scheduled mon update... 2023-11-15T06:38:16.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f802ff60 msgr2=0x7f25f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:16.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f802ff60 0x7f25f8032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f26000096a0 tx=0x7f2600006b40).stop 2023-11-15T06:38:16.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 msgr2=0x7f2610106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:16.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f2610106720 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f2604013510 tx=0x7f260400b040).stop 2023-11-15T06:38:16.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 shutdown_connections 2023-11-15T06:38:16.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f802ff60 0x7f25f8032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:16.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 --2- 172.21.15.135:0/3146717475 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26100f5410 0x7f2610106720 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:16.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 >> 172.21.15.135:0/3146717475 conn(0x7f26100f0b70 msgr2=0x7f261005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:16.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 shutdown_connections 2023-11-15T06:38:16.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:16.765+0000 7f2615a36700 1 -- 172.21.15.135:0/3146717475 wait complete. 2023-11-15T06:38:17.130 DEBUG:teuthology.orchestra.run.smithi135:mon.smithi135> sudo journalctl -f -n 0 -u ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135.service 2023-11-15T06:38:17.133 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:17.133 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:17.162 INFO:journalctl@ceph.mon.smithi135.smithi135.stdout:-- Logs begin at Wed 2023-11-15 06:24:39 UTC. -- 2023-11-15T06:38:18.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:16.235496+0000 mgr.smithi123.razjtn (mgr.14162) 17 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:18.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:16.760430+0000 mgr.smithi123.razjtn (mgr.14162) 18 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135", "target": ["mon-mgr", ""]}]: dispatch 2023-11-15T06:38:18.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:38:16.761774+0000 mgr.smithi123.razjtn (mgr.14162) 19 : cephadm [INF] Saving service mon spec with placement smithi123:172.21.15.123=smithi123;smithi135:172.21.15.135=smithi135;count:2 2023-11-15T06:38:18.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:16.764330+0000 mon.smithi123 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:38:18.277 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 -- 172.21.15.135:0/490351863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 msgr2=0x7fef3c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:18.277 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 --2- 172.21.15.135:0/490351863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c0f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fef30009fb0 tx=0x7fef30009b30).stop 2023-11-15T06:38:18.277 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 -- 172.21.15.135:0/490351863 shutdown_connections 2023-11-15T06:38:18.277 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 --2- 172.21.15.135:0/490351863 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c0f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:18.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 -- 172.21.15.135:0/490351863 >> 172.21.15.135:0/490351863 conn(0x7fef3c0f0b70 msgr2=0x7fef3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:18.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 -- 172.21.15.135:0/490351863 shutdown_connections 2023-11-15T06:38:18.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 -- 172.21.15.135:0/490351863 wait complete. 2023-11-15T06:38:18.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.273+0000 7fef42ab1700 1 Processor -- start 2023-11-15T06:38:18.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef42ab1700 1 -- start start 2023-11-15T06:38:18.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef42ab1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:18.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef42ab1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fef3c106ba0 con 0x7fef3c0f5410 2023-11-15T06:38:18.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef4084d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:18.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef4084d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36650/0 (socket says 172.21.15.135:36650) 2023-11-15T06:38:18.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef4084d700 1 -- 172.21.15.135:0/1927190183 learned_addr learned my addr 172.21.15.135:0/1927190183 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:18.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef4084d700 1 -- 172.21.15.135:0/1927190183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fef30005040 con 0x7fef3c0f5410 2023-11-15T06:38:18.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.274+0000 7fef4084d700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c1066b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fef30013510 tx=0x7fef3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:18.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.275+0000 7fef397fa700 1 -- 172.21.15.135:0/1927190183 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef30009320 con 0x7fef3c0f5410 2023-11-15T06:38:18.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.275+0000 7fef397fa700 1 -- 172.21.15.135:0/1927190183 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fef30013f00 con 0x7fef3c0f5410 2023-11-15T06:38:18.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.275+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fef3c106da0 con 0x7fef3c0f5410 2023-11-15T06:38:18.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.275+0000 7fef397fa700 1 -- 172.21.15.135:0/1927190183 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fef30014d90 con 0x7fef3c0f5410 2023-11-15T06:38:18.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.275+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fef3c107230 con 0x7fef3c0f5410 2023-11-15T06:38:18.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.275+0000 7fef397fa700 1 -- 172.21.15.135:0/1927190183 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fef300263a0 con 0x7fef3c0f5410 2023-11-15T06:38:18.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.275+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fef3c045ad0 con 0x7fef3c0f5410 2023-11-15T06:38:18.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.276+0000 7fef397fa700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fef2c02ff60 0x7fef2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:18.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.276+0000 7fef397fa700 1 -- 172.21.15.135:0/1927190183 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fef30041070 con 0x7fef3c0f5410 2023-11-15T06:38:18.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.276+0000 7fef3bfff700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fef2c02ff60 0x7fef2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:18.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.276+0000 7fef3bfff700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fef2c02ff60 0x7fef2c032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fef280096a0 tx=0x7fef28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:18.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.280+0000 7fef397fa700 1 -- 172.21.15.135:0/1927190183 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fef300095f0 con 0x7fef3c0f5410 2023-11-15T06:38:18.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.485+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fef3c1080e0 con 0x7fef3c0f5410 2023-11-15T06:38:18.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.485+0000 7fef397fa700 1 -- 172.21.15.135:0/1927190183 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fef3001db70 con 0x7fef3c0f5410 2023-11-15T06:38:18.488 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:18.488 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:18.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fef2c02ff60 msgr2=0x7fef2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:18.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fef2c02ff60 0x7fef2c032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fef280096a0 tx=0x7fef28006b40).stop 2023-11-15T06:38:18.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 msgr2=0x7fef3c1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:18.491 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c1066b0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fef30013510 tx=0x7fef3000b040).stop 2023-11-15T06:38:18.491 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 shutdown_connections 2023-11-15T06:38:18.491 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fef2c02ff60 0x7fef2c032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:18.491 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 --2- 172.21.15.135:0/1927190183 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fef3c0f5410 0x7fef3c1066b0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:18.491 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 >> 172.21.15.135:0/1927190183 conn(0x7fef3c0f0b70 msgr2=0x7fef3c0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:18.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 shutdown_connections 2023-11-15T06:38:18.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:18.487+0000 7fef42ab1700 1 -- 172.21.15.135:0/1927190183 wait complete. 2023-11-15T06:38:18.492 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:19.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:18 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:18.486330+0000 mon.smithi123 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.135:0/1927190183' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:19.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:19.873 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:20.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:18.235854+0000 mgr.smithi123.razjtn (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:20.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.942+0000 7f8a62b86700 1 -- 172.21.15.135:0/3765279019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 msgr2=0x7f8a5c0fb570 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:20.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.942+0000 7f8a62b86700 1 --2- 172.21.15.135:0/3765279019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c0fb570 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f8a50009fb0 tx=0x7f8a50009b30).stop 2023-11-15T06:38:20.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.942+0000 7f8a62b86700 1 -- 172.21.15.135:0/3765279019 shutdown_connections 2023-11-15T06:38:20.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.942+0000 7f8a62b86700 1 --2- 172.21.15.135:0/3765279019 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c0fb570 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:20.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.942+0000 7f8a62b86700 1 -- 172.21.15.135:0/3765279019 >> 172.21.15.135:0/3765279019 conn(0x7f8a5c0f0b70 msgr2=0x7f8a5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:20.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.942+0000 7f8a62b86700 1 -- 172.21.15.135:0/3765279019 shutdown_connections 2023-11-15T06:38:20.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.942+0000 7f8a62b86700 1 -- 172.21.15.135:0/3765279019 wait complete. 2023-11-15T06:38:20.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a62b86700 1 Processor -- start 2023-11-15T06:38:20.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a62b86700 1 -- start start 2023-11-15T06:38:20.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a62b86700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c1088d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:20.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a62b86700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a5c108dc0 con 0x7f8a5c060200 2023-11-15T06:38:20.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a60922700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c1088d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:20.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a60922700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c1088d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36666/0 (socket says 172.21.15.135:36666) 2023-11-15T06:38:20.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a60922700 1 -- 172.21.15.135:0/83489173 learned_addr learned my addr 172.21.15.135:0/83489173 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:20.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a60922700 1 -- 172.21.15.135:0/83489173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8a50005040 con 0x7f8a5c060200 2023-11-15T06:38:20.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.943+0000 7f8a60922700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c1088d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f8a50013510 tx=0x7f8a5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:20.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a597fa700 1 -- 172.21.15.135:0/83489173 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a50009320 con 0x7f8a5c060200 2023-11-15T06:38:20.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a5c108fc0 con 0x7f8a5c060200 2023-11-15T06:38:20.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a597fa700 1 -- 172.21.15.135:0/83489173 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8a50013f00 con 0x7f8a5c060200 2023-11-15T06:38:20.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a597fa700 1 -- 172.21.15.135:0/83489173 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8a50014dc0 con 0x7f8a5c060200 2023-11-15T06:38:20.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a5c109450 con 0x7f8a5c060200 2023-11-15T06:38:20.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a597fa700 1 -- 172.21.15.135:0/83489173 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8a5001d3d0 con 0x7f8a5c060200 2023-11-15T06:38:20.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a597fa700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a4c02ff60 0x7f8a4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:20.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.945+0000 7f8a597fa700 1 -- 172.21.15.135:0/83489173 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8a50041410 con 0x7f8a5c060200 2023-11-15T06:38:20.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.944+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a5c045ad0 con 0x7f8a5c060200 2023-11-15T06:38:20.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.945+0000 7f8a5bfff700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a4c02ff60 0x7f8a4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:20.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.945+0000 7f8a5bfff700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a4c02ff60 0x7f8a4c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8a480096a0 tx=0x7f8a48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:20.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:20.949+0000 7f8a597fa700 1 -- 172.21.15.135:0/83489173 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8a50012920 con 0x7f8a5c060200 2023-11-15T06:38:21.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.158+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a5c10a300 con 0x7f8a5c060200 2023-11-15T06:38:21.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.159+0000 7f8a597fa700 1 -- 172.21.15.135:0/83489173 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8a5002b030 con 0x7f8a5c060200 2023-11-15T06:38:21.162 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:21.162 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:21.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.160+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a4c02ff60 msgr2=0x7f8a4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:21.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.160+0000 7f8a62b86700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a4c02ff60 0x7f8a4c032420 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f8a480096a0 tx=0x7f8a48006b40).stop 2023-11-15T06:38:21.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.160+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 msgr2=0x7f8a5c1088d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:21.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.160+0000 7f8a62b86700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c1088d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f8a50013510 tx=0x7f8a5000b040).stop 2023-11-15T06:38:21.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.161+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 shutdown_connections 2023-11-15T06:38:21.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.161+0000 7f8a62b86700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8a4c02ff60 0x7f8a4c032420 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:21.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.161+0000 7f8a62b86700 1 --2- 172.21.15.135:0/83489173 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8a5c060200 0x7f8a5c1088d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:21.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.161+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 >> 172.21.15.135:0/83489173 conn(0x7f8a5c0f0b70 msgr2=0x7f8a5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:21.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.161+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 shutdown_connections 2023-11-15T06:38:21.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:21.161+0000 7f8a62b86700 1 -- 172.21.15.135:0/83489173 wait complete. 2023-11-15T06:38:21.165 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:22.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:20.236084+0000 mgr.smithi123.razjtn (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:22.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:21.160007+0000 mon.smithi123 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.135:0/83489173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:22.528 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:22.528 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:23.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 -- 172.21.15.135:0/2025304302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 msgr2=0x7fc4440f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:23.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 --2- 172.21.15.135:0/2025304302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f7d00 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7fc434009fb0 tx=0x7fc434009b30).stop 2023-11-15T06:38:23.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 -- 172.21.15.135:0/2025304302 shutdown_connections 2023-11-15T06:38:23.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 --2- 172.21.15.135:0/2025304302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f7d00 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:23.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 -- 172.21.15.135:0/2025304302 >> 172.21.15.135:0/2025304302 conn(0x7fc4440f0120 msgr2=0x7fc4440f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:23.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 -- 172.21.15.135:0/2025304302 shutdown_connections 2023-11-15T06:38:23.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 -- 172.21.15.135:0/2025304302 wait complete. 2023-11-15T06:38:23.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 Processor -- start 2023-11-15T06:38:23.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.677+0000 7fc443fff700 1 -- start start 2023-11-15T06:38:23.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc443fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f71b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:23.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc443fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc4440f5630 con 0x7fc4440f78e0 2023-11-15T06:38:23.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc442ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f71b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:23.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc442ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f71b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36684/0 (socket says 172.21.15.135:36684) 2023-11-15T06:38:23.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc442ffd700 1 -- 172.21.15.135:0/3984116857 learned_addr learned my addr 172.21.15.135:0/3984116857 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:23.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc442ffd700 1 -- 172.21.15.135:0/3984116857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc434005040 con 0x7fc4440f78e0 2023-11-15T06:38:23.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc442ffd700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f71b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc434013510 tx=0x7fc43400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:23.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc42b7fe700 1 -- 172.21.15.135:0/3984116857 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc434009320 con 0x7fc4440f78e0 2023-11-15T06:38:23.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc4440f57d0 con 0x7fc4440f78e0 2023-11-15T06:38:23.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc42b7fe700 1 -- 172.21.15.135:0/3984116857 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc434013f00 con 0x7fc4440f78e0 2023-11-15T06:38:23.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc42b7fe700 1 -- 172.21.15.135:0/3984116857 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc434014dc0 con 0x7fc4440f78e0 2023-11-15T06:38:23.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.678+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc4440f5c60 con 0x7fc4440f78e0 2023-11-15T06:38:23.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.679+0000 7fc42b7fe700 1 -- 172.21.15.135:0/3984116857 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc43401d3d0 con 0x7fc4440f78e0 2023-11-15T06:38:23.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.679+0000 7fc42b7fe700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc42c02ff60 0x7fc42c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:23.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.679+0000 7fc42b7fe700 1 -- 172.21.15.135:0/3984116857 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc4340410d0 con 0x7fc4440f78e0 2023-11-15T06:38:23.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.679+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc444059100 con 0x7fc4440f78e0 2023-11-15T06:38:23.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.679+0000 7fc4427fc700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc42c02ff60 0x7fc42c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:23.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.680+0000 7fc4427fc700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc42c02ff60 0x7fc42c032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc4380096a0 tx=0x7fc438006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:23.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.684+0000 7fc42b7fe700 1 -- 172.21.15.135:0/3984116857 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc4340095f0 con 0x7fc4440f78e0 2023-11-15T06:38:23.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.890+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc4440f6360 con 0x7fc4440f78e0 2023-11-15T06:38:23.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.891+0000 7fc42b7fe700 1 -- 172.21.15.135:0/3984116857 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc43402b020 con 0x7fc4440f78e0 2023-11-15T06:38:23.894 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:23.894 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:23.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc42c02ff60 msgr2=0x7fc42c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:23.896 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc42c02ff60 0x7fc42c032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7fc4380096a0 tx=0x7fc438006b40).stop 2023-11-15T06:38:23.896 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 msgr2=0x7fc4440f71b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:23.896 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f71b0 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fc434013510 tx=0x7fc43400b040).stop 2023-11-15T06:38:23.896 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 shutdown_connections 2023-11-15T06:38:23.896 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc42c02ff60 0x7fc42c032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:23.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 --2- 172.21.15.135:0/3984116857 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc4440f78e0 0x7fc4440f71b0 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:23.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 >> 172.21.15.135:0/3984116857 conn(0x7fc4440f0120 msgr2=0x7fc4440f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:23.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 shutdown_connections 2023-11-15T06:38:23.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:23.893+0000 7fc443fff700 1 -- 172.21.15.135:0/3984116857 wait complete. 2023-11-15T06:38:23.898 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:24.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:22.236401+0000 mgr.smithi123.razjtn (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:25.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:24 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:23.892100+0000 mon.smithi123 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.135:0/3984116857' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:25.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:25.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:26.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:24.236679+0000 mgr.smithi123.razjtn (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:26.396 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.393+0000 7feb4962b700 1 -- 172.21.15.135:0/3884527054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 msgr2=0x7feb440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:26.396 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.393+0000 7feb4962b700 1 --2- 172.21.15.135:0/3884527054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb440f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7feb38009fb0 tx=0x7feb38009b30).stop 2023-11-15T06:38:26.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.393+0000 7feb4962b700 1 -- 172.21.15.135:0/3884527054 shutdown_connections 2023-11-15T06:38:26.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.393+0000 7feb4962b700 1 --2- 172.21.15.135:0/3884527054 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb440f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:26.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.393+0000 7feb4962b700 1 -- 172.21.15.135:0/3884527054 >> 172.21.15.135:0/3884527054 conn(0x7feb440f0b70 msgr2=0x7feb440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:26.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.393+0000 7feb4962b700 1 -- 172.21.15.135:0/3884527054 shutdown_connections 2023-11-15T06:38:26.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.393+0000 7feb4962b700 1 -- 172.21.15.135:0/3884527054 wait complete. 2023-11-15T06:38:26.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.394+0000 7feb4962b700 1 Processor -- start 2023-11-15T06:38:26.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.394+0000 7feb4962b700 1 -- start start 2023-11-15T06:38:26.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.394+0000 7feb4962b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb44105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:26.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.394+0000 7feb4962b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb44105b20 con 0x7feb440f5410 2023-11-15T06:38:26.402 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb42ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb44105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:26.403 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb42ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb44105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36704/0 (socket says 172.21.15.135:36704) 2023-11-15T06:38:26.403 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb42ffd700 1 -- 172.21.15.135:0/264802091 learned_addr learned my addr 172.21.15.135:0/264802091 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:26.403 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb42ffd700 1 -- 172.21.15.135:0/264802091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb38005040 con 0x7feb440f5410 2023-11-15T06:38:26.403 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb42ffd700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb44105630 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7feb38013510 tx=0x7feb3800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:26.403 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb2b7fe700 1 -- 172.21.15.135:0/264802091 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb38009320 con 0x7feb440f5410 2023-11-15T06:38:26.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb2b7fe700 1 -- 172.21.15.135:0/264802091 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb38013f00 con 0x7feb440f5410 2023-11-15T06:38:26.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb44105d20 con 0x7feb440f5410 2023-11-15T06:38:26.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb2b7fe700 1 -- 172.21.15.135:0/264802091 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb38014d90 con 0x7feb440f5410 2023-11-15T06:38:26.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.395+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb441061b0 con 0x7feb440f5410 2023-11-15T06:38:26.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.396+0000 7feb2b7fe700 1 -- 172.21.15.135:0/264802091 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb380263a0 con 0x7feb440f5410 2023-11-15T06:38:26.405 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.396+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb44045ad0 con 0x7feb440f5410 2023-11-15T06:38:26.405 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.396+0000 7feb2b7fe700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7feb2c02ffb0 0x7feb2c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:26.405 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.396+0000 7feb2b7fe700 1 -- 172.21.15.135:0/264802091 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb380410f0 con 0x7feb440f5410 2023-11-15T06:38:26.405 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.396+0000 7feb427fc700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7feb2c02ffb0 0x7feb2c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:26.406 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.397+0000 7feb427fc700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7feb2c02ffb0 0x7feb2c032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7feb340096a0 tx=0x7feb34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:26.406 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.401+0000 7feb2b7fe700 1 -- 172.21.15.135:0/264802091 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb380126f0 con 0x7feb440f5410 2023-11-15T06:38:26.610 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.608+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb440fae60 con 0x7feb440f5410 2023-11-15T06:38:26.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.609+0000 7feb2b7fe700 1 -- 172.21.15.135:0/264802091 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb38012330 con 0x7feb440f5410 2023-11-15T06:38:26.612 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:26.612 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:26.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7feb2c02ffb0 msgr2=0x7feb2c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:26.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7feb2c02ffb0 0x7feb2c032470 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7feb340096a0 tx=0x7feb34006b40).stop 2023-11-15T06:38:26.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 msgr2=0x7feb44105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:26.614 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb44105630 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7feb38013510 tx=0x7feb3800b040).stop 2023-11-15T06:38:26.614 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 shutdown_connections 2023-11-15T06:38:26.614 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7feb2c02ffb0 0x7feb2c032470 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:26.614 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 --2- 172.21.15.135:0/264802091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feb440f5410 0x7feb44105630 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:26.614 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 >> 172.21.15.135:0/264802091 conn(0x7feb440f0b70 msgr2=0x7feb440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:26.615 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.610+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 shutdown_connections 2023-11-15T06:38:26.615 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:26.611+0000 7feb4962b700 1 -- 172.21.15.135:0/264802091 wait complete. 2023-11-15T06:38:26.615 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:27.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:26 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:26.609715+0000 mon.smithi123 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.135:0/264802091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:27.997 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:27.998 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:28.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:26.237058+0000 mgr.smithi123.razjtn (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:29.082 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.078+0000 7f00ce0eb700 1 -- 172.21.15.135:0/588230883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 msgr2=0x7f00c80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:29.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.078+0000 7f00ce0eb700 1 --2- 172.21.15.135:0/588230883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c80f90f0 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f00b8009fb0 tx=0x7f00b8009b30).stop 2023-11-15T06:38:29.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 -- 172.21.15.135:0/588230883 shutdown_connections 2023-11-15T06:38:29.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 --2- 172.21.15.135:0/588230883 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c80f90f0 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:29.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 -- 172.21.15.135:0/588230883 >> 172.21.15.135:0/588230883 conn(0x7f00c80f0b70 msgr2=0x7f00c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:29.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 -- 172.21.15.135:0/588230883 shutdown_connections 2023-11-15T06:38:29.084 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 -- 172.21.15.135:0/588230883 wait complete. 2023-11-15T06:38:29.084 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 Processor -- start 2023-11-15T06:38:29.084 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 -- start start 2023-11-15T06:38:29.084 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c81088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:29.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.079+0000 7f00ce0eb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f00c8108de0 con 0x7f00c8060280 2023-11-15T06:38:29.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00c77fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c81088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:29.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00c77fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c81088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:38876/0 (socket says 172.21.15.135:38876) 2023-11-15T06:38:29.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00c77fe700 1 -- 172.21.15.135:0/158561598 learned_addr learned my addr 172.21.15.135:0/158561598 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:29.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00c77fe700 1 -- 172.21.15.135:0/158561598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f00b8005040 con 0x7f00c8060280 2023-11-15T06:38:29.086 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00c77fe700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c81088f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f00b8013510 tx=0x7f00b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:29.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00affff700 1 -- 172.21.15.135:0/158561598 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00b8009320 con 0x7f00c8060280 2023-11-15T06:38:29.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f00c8108fe0 con 0x7f00c8060280 2023-11-15T06:38:29.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f00c8109470 con 0x7f00c8060280 2023-11-15T06:38:29.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00affff700 1 -- 172.21.15.135:0/158561598 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f00b8013f00 con 0x7f00c8060280 2023-11-15T06:38:29.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.080+0000 7f00affff700 1 -- 172.21.15.135:0/158561598 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f00b8014d90 con 0x7f00c8060280 2023-11-15T06:38:29.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.081+0000 7f00affff700 1 -- 172.21.15.135:0/158561598 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f00b80263a0 con 0x7f00c8060280 2023-11-15T06:38:29.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.081+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f00a8004fa0 con 0x7f00c8060280 2023-11-15T06:38:29.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.081+0000 7f00affff700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f00b002ff60 0x7f00b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:29.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.081+0000 7f00affff700 1 -- 172.21.15.135:0/158561598 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f00b8041290 con 0x7f00c8060280 2023-11-15T06:38:29.089 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.081+0000 7f00c6ffd700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f00b002ff60 0x7f00b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:29.089 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.082+0000 7f00c6ffd700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f00b002ff60 0x7f00b0032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f00bc0096a0 tx=0x7f00bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:29.089 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.086+0000 7f00affff700 1 -- 172.21.15.135:0/158561598 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f00b8012330 con 0x7f00c8060280 2023-11-15T06:38:29.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.289+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f00a8005670 con 0x7f00c8060280 2023-11-15T06:38:29.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.290+0000 7f00affff700 1 -- 172.21.15.135:0/158561598 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f00b801cc70 con 0x7f00c8060280 2023-11-15T06:38:29.293 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:29.294 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:29.295 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f00b002ff60 msgr2=0x7f00b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:29.295 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f00b002ff60 0x7f00b0032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f00bc0096a0 tx=0x7f00bc006b40).stop 2023-11-15T06:38:29.295 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 msgr2=0x7f00c81088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:29.295 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c81088f0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f00b8013510 tx=0x7f00b800b040).stop 2023-11-15T06:38:29.295 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 shutdown_connections 2023-11-15T06:38:29.296 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f00b002ff60 0x7f00b0032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:29.296 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 --2- 172.21.15.135:0/158561598 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f00c8060280 0x7f00c81088f0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:29.296 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 >> 172.21.15.135:0/158561598 conn(0x7f00c80f0b70 msgr2=0x7f00c80f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:29.296 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 shutdown_connections 2023-11-15T06:38:29.296 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:29.292+0000 7f00ce0eb700 1 -- 172.21.15.135:0/158561598 wait complete. 2023-11-15T06:38:29.297 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:30.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:28.237417+0000 mgr.smithi123.razjtn (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:30.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:29.291080+0000 mon.smithi123 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.135:0/158561598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:30.652 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:30.652 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:31.752 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 -- 172.21.15.135:0/2095552060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 msgr2=0x7fa7a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:31.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 --2- 172.21.15.135:0/2095552060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a40f5830 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fa798009fb0 tx=0x7fa798009b30).stop 2023-11-15T06:38:31.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 -- 172.21.15.135:0/2095552060 shutdown_connections 2023-11-15T06:38:31.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 --2- 172.21.15.135:0/2095552060 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a40f5830 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:31.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 -- 172.21.15.135:0/2095552060 >> 172.21.15.135:0/2095552060 conn(0x7fa7a40f0b70 msgr2=0x7fa7a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:31.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 -- 172.21.15.135:0/2095552060 shutdown_connections 2023-11-15T06:38:31.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 -- 172.21.15.135:0/2095552060 wait complete. 2023-11-15T06:38:31.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.749+0000 7fa7aa499700 1 Processor -- start 2023-11-15T06:38:31.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.750+0000 7fa7aa499700 1 -- start start 2023-11-15T06:38:31.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.750+0000 7fa7aa499700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a41093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:31.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.750+0000 7fa7aa499700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7a4060240 con 0x7fa7a40f5410 2023-11-15T06:38:31.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.750+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a41093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:31.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.750+0000 7fa7a3fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a41093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:38902/0 (socket says 172.21.15.135:38902) 2023-11-15T06:38:31.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.750+0000 7fa7a3fff700 1 -- 172.21.15.135:0/2708575100 learned_addr learned my addr 172.21.15.135:0/2708575100 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:31.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.751+0000 7fa7a3fff700 1 -- 172.21.15.135:0/2708575100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa798005040 con 0x7fa7a40f5410 2023-11-15T06:38:31.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.751+0000 7fa7a3fff700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a41093d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fa798013510 tx=0x7fa79800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:31.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.751+0000 7fa7a0ff9700 1 -- 172.21.15.135:0/2708575100 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa798009320 con 0x7fa7a40f5410 2023-11-15T06:38:31.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.751+0000 7fa7a0ff9700 1 -- 172.21.15.135:0/2708575100 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa798013f00 con 0x7fa7a40f5410 2023-11-15T06:38:31.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.751+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7a4060440 con 0x7fa7a40f5410 2023-11-15T06:38:31.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.751+0000 7fa7a0ff9700 1 -- 172.21.15.135:0/2708575100 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa798014d90 con 0x7fa7a40f5410 2023-11-15T06:38:31.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.751+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7a4109a80 con 0x7fa7a40f5410 2023-11-15T06:38:31.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.752+0000 7fa7a0ff9700 1 -- 172.21.15.135:0/2708575100 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7980263a0 con 0x7fa7a40f5410 2023-11-15T06:38:31.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.752+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7a4045ad0 con 0x7fa7a40f5410 2023-11-15T06:38:31.760 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.752+0000 7fa7a0ff9700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa78c02ffb0 0x7fa78c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:31.760 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.752+0000 7fa7a0ff9700 1 -- 172.21.15.135:0/2708575100 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7980412b0 con 0x7fa7a40f5410 2023-11-15T06:38:31.760 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.752+0000 7fa7a37fe700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa78c02ffb0 0x7fa78c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:31.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.752+0000 7fa7a37fe700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa78c02ffb0 0x7fa78c032470 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fa7940096a0 tx=0x7fa794006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:31.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.756+0000 7fa7a0ff9700 1 -- 172.21.15.135:0/2708575100 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa798012720 con 0x7fa7a40f5410 2023-11-15T06:38:31.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.964+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7a410a290 con 0x7fa7a40f5410 2023-11-15T06:38:31.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.965+0000 7fa7a0ff9700 1 -- 172.21.15.135:0/2708575100 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa798012330 con 0x7fa7a40f5410 2023-11-15T06:38:31.968 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:31.968 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:31.973 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.966+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa78c02ffb0 msgr2=0x7fa78c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:31.973 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.966+0000 7fa7aa499700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa78c02ffb0 0x7fa78c032470 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7fa7940096a0 tx=0x7fa794006b40).stop 2023-11-15T06:38:31.973 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.966+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 msgr2=0x7fa7a41093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:31.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.966+0000 7fa7aa499700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a41093d0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7fa798013510 tx=0x7fa79800b040).stop 2023-11-15T06:38:31.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.966+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 shutdown_connections 2023-11-15T06:38:31.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.967+0000 7fa7aa499700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa78c02ffb0 0x7fa78c032470 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:31.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.967+0000 7fa7aa499700 1 --2- 172.21.15.135:0/2708575100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa7a40f5410 0x7fa7a41093d0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:31.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.967+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 >> 172.21.15.135:0/2708575100 conn(0x7fa7a40f0b70 msgr2=0x7fa7a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:31.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.967+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 shutdown_connections 2023-11-15T06:38:31.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:31.967+0000 7fa7aa499700 1 -- 172.21.15.135:0/2708575100 wait complete. 2023-11-15T06:38:31.975 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:32.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:30.237659+0000 mgr.smithi123.razjtn (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:33.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:31.966016+0000 mon.smithi123 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.135:0/2708575100' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:33.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:33.311 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:34.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:32.238018+0000 mgr.smithi123.razjtn (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:34.395 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 -- 172.21.15.135:0/3615882177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 msgr2=0x7efcc80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:34.395 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 --2- 172.21.15.135:0/3615882177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80f55c0 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7efcb8009fb0 tx=0x7efcb8009b30).stop 2023-11-15T06:38:34.396 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 -- 172.21.15.135:0/3615882177 shutdown_connections 2023-11-15T06:38:34.396 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 --2- 172.21.15.135:0/3615882177 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80f55c0 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:34.396 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 -- 172.21.15.135:0/3615882177 >> 172.21.15.135:0/3615882177 conn(0x7efcc80f09a0 msgr2=0x7efcc80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:34.396 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 -- 172.21.15.135:0/3615882177 shutdown_connections 2023-11-15T06:38:34.396 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 -- 172.21.15.135:0/3615882177 wait complete. 2023-11-15T06:38:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 Processor -- start 2023-11-15T06:38:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.391+0000 7efccd107700 1 -- start start 2023-11-15T06:38:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efccd107700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efccd107700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efcc80fb5a0 con 0x7efcc80f51a0 2023-11-15T06:38:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:34.398 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:38926/0 (socket says 172.21.15.135:38926) 2023-11-15T06:38:34.398 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc7fff700 1 -- 172.21.15.135:0/3994647121 learned_addr learned my addr 172.21.15.135:0/3994647121 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:34.398 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc7fff700 1 -- 172.21.15.135:0/3994647121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efcb8005040 con 0x7efcc80f51a0 2023-11-15T06:38:34.398 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc7fff700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80fd120 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efcb8013510 tx=0x7efcb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:34.398 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc4ff9700 1 -- 172.21.15.135:0/3994647121 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcb8009320 con 0x7efcc80f51a0 2023-11-15T06:38:34.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efcc80fb740 con 0x7efcc80f51a0 2023-11-15T06:38:34.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc4ff9700 1 -- 172.21.15.135:0/3994647121 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efcb8013f00 con 0x7efcc80f51a0 2023-11-15T06:38:34.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.392+0000 7efcc4ff9700 1 -- 172.21.15.135:0/3994647121 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efcb8014dc0 con 0x7efcc80f51a0 2023-11-15T06:38:34.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.393+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efcc80fbbd0 con 0x7efcc80f51a0 2023-11-15T06:38:34.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.393+0000 7efcc4ff9700 1 -- 172.21.15.135:0/3994647121 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efcb8014420 con 0x7efcc80f51a0 2023-11-15T06:38:34.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.393+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efcc8059100 con 0x7efcc80f51a0 2023-11-15T06:38:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.393+0000 7efcc4ff9700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efcb002ffb0 0x7efcb0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.393+0000 7efcc4ff9700 1 -- 172.21.15.135:0/3994647121 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efcb80410b0 con 0x7efcc80f51a0 2023-11-15T06:38:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.394+0000 7efcc77fe700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efcb002ffb0 0x7efcb0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.394+0000 7efcc77fe700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efcb002ffb0 0x7efcb0032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7efcbc0096a0 tx=0x7efcbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.398+0000 7efcc4ff9700 1 -- 172.21.15.135:0/3994647121 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efcb806d2b0 con 0x7efcc80f51a0 2023-11-15T06:38:34.609 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.607+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efcc80fabe0 con 0x7efcc80f51a0 2023-11-15T06:38:34.610 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.607+0000 7efcc4ff9700 1 -- 172.21.15.135:0/3994647121 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efcb80095f0 con 0x7efcc80f51a0 2023-11-15T06:38:34.610 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:34.610 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:34.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efcb002ffb0 msgr2=0x7efcb0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:34.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efcb002ffb0 0x7efcb0032470 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7efcbc0096a0 tx=0x7efcbc006b40).stop 2023-11-15T06:38:34.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 msgr2=0x7efcc80fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:34.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80fd120 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7efcb8013510 tx=0x7efcb800b040).stop 2023-11-15T06:38:34.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 shutdown_connections 2023-11-15T06:38:34.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efcb002ffb0 0x7efcb0032470 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:34.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 --2- 172.21.15.135:0/3994647121 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efcc80f51a0 0x7efcc80fd120 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:34.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 >> 172.21.15.135:0/3994647121 conn(0x7efcc80f09a0 msgr2=0x7efcc805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:34.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 shutdown_connections 2023-11-15T06:38:34.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:34.609+0000 7efccd107700 1 -- 172.21.15.135:0/3994647121 wait complete. 2023-11-15T06:38:34.614 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:35.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:34 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:34.608334+0000 mon.smithi123 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.135:0/3994647121' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:35.996 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:35.997 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:36.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:34.238390+0000 mgr.smithi123.razjtn (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:38.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:36.238617+0000 mgr.smithi123.razjtn (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:40.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:38.238899+0000 mgr.smithi123.razjtn (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:41.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:40.362542+0000 mon.smithi123 (mon.0) 128 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:38:41.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:40.369378+0000 mon.smithi123 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:38:41.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.266+0000 7f83efb69700 1 -- 172.21.15.135:0/3682739566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 msgr2=0x7f83e80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:41.281 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.266+0000 7f83efb69700 1 --2- 172.21.15.135:0/3682739566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e80f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f83d8009fb0 tx=0x7f83d8009b30).stop 2023-11-15T06:38:41.281 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 -- 172.21.15.135:0/3682739566 shutdown_connections 2023-11-15T06:38:41.281 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 --2- 172.21.15.135:0/3682739566 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e80f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:41.281 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 -- 172.21.15.135:0/3682739566 >> 172.21.15.135:0/3682739566 conn(0x7f83e80f0b70 msgr2=0x7f83e80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:41.281 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 -- 172.21.15.135:0/3682739566 shutdown_connections 2023-11-15T06:38:41.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 -- 172.21.15.135:0/3682739566 wait complete. 2023-11-15T06:38:41.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 Processor -- start 2023-11-15T06:38:41.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 -- start start 2023-11-15T06:38:41.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:41.282 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83efb69700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f83e8060240 con 0x7f83e80f5410 2023-11-15T06:38:41.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83ed905700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:41.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83ed905700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56110/0 (socket says 172.21.15.135:56110) 2023-11-15T06:38:41.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.267+0000 7f83ed905700 1 -- 172.21.15.135:0/82870495 learned_addr learned my addr 172.21.15.135:0/82870495 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:41.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.268+0000 7f83ed905700 1 -- 172.21.15.135:0/82870495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f83d8005040 con 0x7f83e80f5410 2023-11-15T06:38:41.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.268+0000 7f83ed905700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e8109430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f83d8013510 tx=0x7f83d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:41.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.268+0000 7f83de7fc700 1 -- 172.21.15.135:0/82870495 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d8009320 con 0x7f83e80f5410 2023-11-15T06:38:41.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.268+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f83e8060440 con 0x7f83e80f5410 2023-11-15T06:38:41.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.268+0000 7f83de7fc700 1 -- 172.21.15.135:0/82870495 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f83d8013f00 con 0x7f83e80f5410 2023-11-15T06:38:41.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.268+0000 7f83de7fc700 1 -- 172.21.15.135:0/82870495 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f83d8014d90 con 0x7f83e80f5410 2023-11-15T06:38:41.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.268+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f83e8109ae0 con 0x7f83e80f5410 2023-11-15T06:38:41.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.269+0000 7f83de7fc700 1 -- 172.21.15.135:0/82870495 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f83d80095f0 con 0x7f83e80f5410 2023-11-15T06:38:41.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.269+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f83e8045ad0 con 0x7f83e80f5410 2023-11-15T06:38:41.286 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.269+0000 7f83de7fc700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f83d402ff60 0x7f83d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:41.286 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.269+0000 7f83de7fc700 1 -- 172.21.15.135:0/82870495 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f83d80410c0 con 0x7f83e80f5410 2023-11-15T06:38:41.286 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.269+0000 7f83ed104700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f83d402ff60 0x7f83d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:41.286 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.269+0000 7f83ed104700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f83d402ff60 0x7f83d4032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83e40096a0 tx=0x7f83e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:41.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.274+0000 7f83de7fc700 1 -- 172.21.15.135:0/82870495 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f83d801d7f0 con 0x7f83e80f5410 2023-11-15T06:38:41.485 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.482+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f83e80fae60 con 0x7f83e80f5410 2023-11-15T06:38:41.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.483+0000 7f83de7fc700 1 -- 172.21.15.135:0/82870495 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f83d8010230 con 0x7f83e80f5410 2023-11-15T06:38:41.486 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:41.486 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:41.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f83d402ff60 msgr2=0x7f83d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:41.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f83d402ff60 0x7f83d4032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7f83e40096a0 tx=0x7f83e4006b40).stop 2023-11-15T06:38:41.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 msgr2=0x7f83e8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:41.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e8109430 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f83d8013510 tx=0x7f83d800b040).stop 2023-11-15T06:38:41.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 shutdown_connections 2023-11-15T06:38:41.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f83d402ff60 0x7f83d4032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:41.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 --2- 172.21.15.135:0/82870495 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f83e80f5410 0x7f83e8109430 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:41.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 >> 172.21.15.135:0/82870495 conn(0x7f83e80f0b70 msgr2=0x7f83e80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:41.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 shutdown_connections 2023-11-15T06:38:41.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:41.485+0000 7f83efb69700 1 -- 172.21.15.135:0/82870495 wait complete. 2023-11-15T06:38:41.490 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:42.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:40.239138+0000 mgr.smithi123.razjtn (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:42.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:41.484250+0000 mon.smithi123 (mon.0) 130 : audit [DBG] from='client.? 172.21.15.135:0/82870495' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:44.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:42.239462+0000 mgr.smithi123.razjtn (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:44.719 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:44.720 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:45.871 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.868+0000 7f8d13609700 1 -- 172.21.15.135:0/1024568880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 msgr2=0x7f8d0c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:45.872 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.868+0000 7f8d13609700 1 --2- 172.21.15.135:0/1024568880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c0f90f0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8cfc009fb0 tx=0x7f8cfc009b30).stop 2023-11-15T06:38:45.872 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.868+0000 7f8d13609700 1 -- 172.21.15.135:0/1024568880 shutdown_connections 2023-11-15T06:38:45.872 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.868+0000 7f8d13609700 1 --2- 172.21.15.135:0/1024568880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c0f90f0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:45.872 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.868+0000 7f8d13609700 1 -- 172.21.15.135:0/1024568880 >> 172.21.15.135:0/1024568880 conn(0x7f8d0c0f0b70 msgr2=0x7f8d0c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:45.872 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.868+0000 7f8d13609700 1 -- 172.21.15.135:0/1024568880 shutdown_connections 2023-11-15T06:38:45.873 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.868+0000 7f8d13609700 1 -- 172.21.15.135:0/1024568880 wait complete. 2023-11-15T06:38:45.873 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.869+0000 7f8d13609700 1 Processor -- start 2023-11-15T06:38:45.873 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.869+0000 7f8d13609700 1 -- start start 2023-11-15T06:38:45.875 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.869+0000 7f8d13609700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:45.875 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.869+0000 7f8d13609700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d0c106bd0 con 0x7f8d0c060280 2023-11-15T06:38:45.876 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.869+0000 7f8d113a5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:45.876 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.869+0000 7f8d113a5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56132/0 (socket says 172.21.15.135:56132) 2023-11-15T06:38:45.876 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.869+0000 7f8d113a5700 1 -- 172.21.15.135:0/1880424106 learned_addr learned my addr 172.21.15.135:0/1880424106 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:45.876 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.870+0000 7f8d113a5700 1 -- 172.21.15.135:0/1880424106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cfc005040 con 0x7f8d0c060280 2023-11-15T06:38:45.876 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.870+0000 7f8d113a5700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c1066e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8cfc013510 tx=0x7f8cfc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:45.877 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.870+0000 7f8d01ffb700 1 -- 172.21.15.135:0/1880424106 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cfc009320 con 0x7f8d0c060280 2023-11-15T06:38:45.877 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.870+0000 7f8d01ffb700 1 -- 172.21.15.135:0/1880424106 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cfc013f00 con 0x7f8d0c060280 2023-11-15T06:38:45.877 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.870+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d0c106dd0 con 0x7f8d0c060280 2023-11-15T06:38:45.877 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.870+0000 7f8d01ffb700 1 -- 172.21.15.135:0/1880424106 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cfc014dc0 con 0x7f8d0c060280 2023-11-15T06:38:45.878 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.870+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d0c107260 con 0x7f8d0c060280 2023-11-15T06:38:45.878 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.871+0000 7f8d01ffb700 1 -- 172.21.15.135:0/1880424106 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cfc014420 con 0x7f8d0c060280 2023-11-15T06:38:45.878 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.871+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d0c045ad0 con 0x7f8d0c060280 2023-11-15T06:38:45.878 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.871+0000 7f8d01ffb700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8cf802ffb0 0x7f8cf8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:45.878 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.871+0000 7f8d01ffb700 1 -- 172.21.15.135:0/1880424106 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cfc040ff0 con 0x7f8d0c060280 2023-11-15T06:38:45.879 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.871+0000 7f8d10ba4700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8cf802ffb0 0x7f8cf8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:45.879 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.872+0000 7f8d10ba4700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8cf802ffb0 0x7f8cf8032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8d080096a0 tx=0x7f8d08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:45.879 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:45.876+0000 7f8d01ffb700 1 -- 172.21.15.135:0/1880424106 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cfc06d2b0 con 0x7f8d0c060280 2023-11-15T06:38:46.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.081+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d0c108110 con 0x7f8d0c060280 2023-11-15T06:38:46.085 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.082+0000 7f8d01ffb700 1 -- 172.21.15.135:0/1880424106 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8cfc0095f0 con 0x7f8d0c060280 2023-11-15T06:38:46.085 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:46.086 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:46.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8cf802ffb0 msgr2=0x7f8cf8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:46.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8cf802ffb0 0x7f8cf8032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f8d080096a0 tx=0x7f8d08006b40).stop 2023-11-15T06:38:46.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 msgr2=0x7f8d0c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:46.087 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c1066e0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f8cfc013510 tx=0x7f8cfc00b040).stop 2023-11-15T06:38:46.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 shutdown_connections 2023-11-15T06:38:46.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8cf802ffb0 0x7f8cf8032470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:46.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 --2- 172.21.15.135:0/1880424106 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d0c060280 0x7f8d0c1066e0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:46.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 >> 172.21.15.135:0/1880424106 conn(0x7f8d0c0f0b70 msgr2=0x7f8d0c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:46.088 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 shutdown_connections 2023-11-15T06:38:46.089 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:46.084+0000 7f8d13609700 1 -- 172.21.15.135:0/1880424106 wait complete. 2023-11-15T06:38:46.089 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:46.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:44.239820+0000 mgr.smithi123.razjtn (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:47.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:46 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:46.083198+0000 mon.smithi123 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.135:0/1880424106' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:47.457 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:47.457 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:48.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:46.240054+0000 mgr.smithi123.razjtn (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:48.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 -- 172.21.15.135:0/110848629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 msgr2=0x7f90a80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:48.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 --2- 172.21.15.135:0/110848629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a80f6930 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f909c009fb0 tx=0x7f909c009b30).stop 2023-11-15T06:38:48.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 -- 172.21.15.135:0/110848629 shutdown_connections 2023-11-15T06:38:48.567 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 --2- 172.21.15.135:0/110848629 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a80f6930 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:48.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 -- 172.21.15.135:0/110848629 >> 172.21.15.135:0/110848629 conn(0x7f90a80f1c70 msgr2=0x7f90a80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:48.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 -- 172.21.15.135:0/110848629 shutdown_connections 2023-11-15T06:38:48.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 -- 172.21.15.135:0/110848629 wait complete. 2023-11-15T06:38:48.568 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.564+0000 7f90ae865700 1 Processor -- start 2023-11-15T06:38:48.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90ae865700 1 -- start start 2023-11-15T06:38:48.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90ae865700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:48.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90ae865700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f90a8069e20 con 0x7f90a80f6510 2023-11-15T06:38:48.570 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:48.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90a7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:52794/0 (socket says 172.21.15.135:52794) 2023-11-15T06:38:48.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90a7fff700 1 -- 172.21.15.135:0/493867133 learned_addr learned my addr 172.21.15.135:0/493867133 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:48.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90a7fff700 1 -- 172.21.15.135:0/493867133 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f909c005040 con 0x7f90a80f6510 2023-11-15T06:38:48.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.565+0000 7f90a7fff700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a806b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f909c013510 tx=0x7f909c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:48.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.566+0000 7f90a4ff9700 1 -- 172.21.15.135:0/493867133 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f909c009320 con 0x7f90a80f6510 2023-11-15T06:38:48.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.566+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f90a8069fc0 con 0x7f90a80f6510 2023-11-15T06:38:48.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.566+0000 7f90a4ff9700 1 -- 172.21.15.135:0/493867133 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f909c013f00 con 0x7f90a80f6510 2023-11-15T06:38:48.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.566+0000 7f90a4ff9700 1 -- 172.21.15.135:0/493867133 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f909c014d90 con 0x7f90a80f6510 2023-11-15T06:38:48.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.566+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f90a806a450 con 0x7f90a80f6510 2023-11-15T06:38:48.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.566+0000 7f90a4ff9700 1 -- 172.21.15.135:0/493867133 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f909c014420 con 0x7f90a80f6510 2023-11-15T06:38:48.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.566+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f90a8059100 con 0x7f90a80f6510 2023-11-15T06:38:48.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.567+0000 7f90a4ff9700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f909002ff60 0x7f9090032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:48.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.567+0000 7f90a4ff9700 1 -- 172.21.15.135:0/493867133 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f909c040fb0 con 0x7f90a80f6510 2023-11-15T06:38:48.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.567+0000 7f90a77fe700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f909002ff60 0x7f9090032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:48.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.567+0000 7f90a77fe700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f909002ff60 0x7f9090032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f90980096a0 tx=0x7f9098006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:48.574 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.571+0000 7f90a4ff9700 1 -- 172.21.15.135:0/493867133 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f909c026bf0 con 0x7f90a80f6510 2023-11-15T06:38:48.779 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.777+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f90a80fbf60 con 0x7f90a80f6510 2023-11-15T06:38:48.780 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.778+0000 7f90a4ff9700 1 -- 172.21.15.135:0/493867133 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f909c01db70 con 0x7f90a80f6510 2023-11-15T06:38:48.780 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:48.781 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:48.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f909002ff60 msgr2=0x7f9090032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:48.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f909002ff60 0x7f9090032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f90980096a0 tx=0x7f9098006b40).stop 2023-11-15T06:38:48.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 msgr2=0x7f90a806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:48.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a806b9a0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f909c013510 tx=0x7f909c00b040).stop 2023-11-15T06:38:48.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 shutdown_connections 2023-11-15T06:38:48.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f909002ff60 0x7f9090032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:48.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 --2- 172.21.15.135:0/493867133 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f90a80f6510 0x7f90a806b9a0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:48.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 >> 172.21.15.135:0/493867133 conn(0x7f90a80f1c70 msgr2=0x7f90a80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:48.784 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 shutdown_connections 2023-11-15T06:38:48.784 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:48.779+0000 7f90ae865700 1 -- 172.21.15.135:0/493867133 wait complete. 2023-11-15T06:38:48.784 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:49.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:48 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:48.778764+0000 mon.smithi123 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.135:0/493867133' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:50.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:48.240374+0000 mgr.smithi123.razjtn (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:50.161 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:50.162 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:51.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 -- 172.21.15.135:0/885643619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 msgr2=0x7f0d180f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:51.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 --2- 172.21.15.135:0/885643619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d180f55c0 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0d0c009fb0 tx=0x7f0d0c009b30).stop 2023-11-15T06:38:51.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 -- 172.21.15.135:0/885643619 shutdown_connections 2023-11-15T06:38:51.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 --2- 172.21.15.135:0/885643619 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d180f55c0 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:51.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 -- 172.21.15.135:0/885643619 >> 172.21.15.135:0/885643619 conn(0x7f0d180f09a0 msgr2=0x7f0d180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:51.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 -- 172.21.15.135:0/885643619 shutdown_connections 2023-11-15T06:38:51.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 -- 172.21.15.135:0/885643619 wait complete. 2023-11-15T06:38:51.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 Processor -- start 2023-11-15T06:38:51.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 -- start start 2023-11-15T06:38:51.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d181064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:51.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1759e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d181069d0 con 0x7f0d180f51a0 2023-11-15T06:38:51.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d181064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:51.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d181064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:52816/0 (socket says 172.21.15.135:52816) 2023-11-15T06:38:51.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.235+0000 7f0d1659c700 1 -- 172.21.15.135:0/1659600517 learned_addr learned my addr 172.21.15.135:0/1659600517 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:51.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.236+0000 7f0d1659c700 1 -- 172.21.15.135:0/1659600517 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d0c005040 con 0x7f0d180f51a0 2023-11-15T06:38:51.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.236+0000 7f0d1659c700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d181064e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0d0c013510 tx=0x7f0d0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:51.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.236+0000 7f0d06ffd700 1 -- 172.21.15.135:0/1659600517 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d0c009320 con 0x7f0d180f51a0 2023-11-15T06:38:51.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.236+0000 7f0d06ffd700 1 -- 172.21.15.135:0/1659600517 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d0c013f00 con 0x7f0d180f51a0 2023-11-15T06:38:51.243 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.236+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d18106bd0 con 0x7f0d180f51a0 2023-11-15T06:38:51.243 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.236+0000 7f0d06ffd700 1 -- 172.21.15.135:0/1659600517 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d0c014dc0 con 0x7f0d180f51a0 2023-11-15T06:38:51.243 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.236+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d18107060 con 0x7f0d180f51a0 2023-11-15T06:38:51.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.237+0000 7f0d06ffd700 1 -- 172.21.15.135:0/1659600517 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d0c014420 con 0x7f0d180f51a0 2023-11-15T06:38:51.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.237+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d18045ad0 con 0x7f0d180f51a0 2023-11-15T06:38:51.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.237+0000 7f0d06ffd700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d0002ffb0 0x7f0d00032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:51.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.237+0000 7f0d06ffd700 1 -- 172.21.15.135:0/1659600517 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d0c040ff0 con 0x7f0d180f51a0 2023-11-15T06:38:51.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.237+0000 7f0d15d9b700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d0002ffb0 0x7f0d00032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:51.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.238+0000 7f0d15d9b700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d0002ffb0 0x7f0d00032470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f0d080096a0 tx=0x7f0d08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:51.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.242+0000 7f0d06ffd700 1 -- 172.21.15.135:0/1659600517 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d0c06d2b0 con 0x7f0d180f51a0 2023-11-15T06:38:51.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.449+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d18107f10 con 0x7f0d180f51a0 2023-11-15T06:38:51.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.450+0000 7f0d06ffd700 1 -- 172.21.15.135:0/1659600517 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0d0c0095f0 con 0x7f0d180f51a0 2023-11-15T06:38:51.453 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:51.453 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:51.454 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.451+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d0002ffb0 msgr2=0x7f0d00032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:51.454 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.451+0000 7f0d1759e700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d0002ffb0 0x7f0d00032470 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f0d080096a0 tx=0x7f0d08006b40).stop 2023-11-15T06:38:51.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.451+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 msgr2=0x7f0d181064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:51.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.451+0000 7f0d1759e700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d181064e0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0d0c013510 tx=0x7f0d0c00b040).stop 2023-11-15T06:38:51.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.452+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 shutdown_connections 2023-11-15T06:38:51.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.452+0000 7f0d1759e700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d0002ffb0 0x7f0d00032470 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:51.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.452+0000 7f0d1759e700 1 --2- 172.21.15.135:0/1659600517 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d180f51a0 0x7f0d181064e0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:51.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.452+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 >> 172.21.15.135:0/1659600517 conn(0x7f0d180f09a0 msgr2=0x7f0d1805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:51.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.452+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 shutdown_connections 2023-11-15T06:38:51.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:51.452+0000 7f0d1759e700 1 -- 172.21.15.135:0/1659600517 wait complete. 2023-11-15T06:38:51.456 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:52.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:50.240675+0000 mgr.smithi123.razjtn (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:52.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:51.450900+0000 mon.smithi123 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.135:0/1659600517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:52.876 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:52.876 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:53.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 -- 172.21.15.135:0/954301240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 msgr2=0x7f23200f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:53.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 --2- 172.21.15.135:0/954301240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f23200f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f2310009fb0 tx=0x7f2310009b30).stop 2023-11-15T06:38:53.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 -- 172.21.15.135:0/954301240 shutdown_connections 2023-11-15T06:38:53.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 --2- 172.21.15.135:0/954301240 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f23200f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:53.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 -- 172.21.15.135:0/954301240 >> 172.21.15.135:0/954301240 conn(0x7f23200f0b70 msgr2=0x7f23200f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:53.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 -- 172.21.15.135:0/954301240 shutdown_connections 2023-11-15T06:38:53.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 -- 172.21.15.135:0/954301240 wait complete. 2023-11-15T06:38:53.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 Processor -- start 2023-11-15T06:38:53.984 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 -- start start 2023-11-15T06:38:53.984 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f232010b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:53.984 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.979+0000 7f2325b33700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2320107240 con 0x7f23200f5410 2023-11-15T06:38:53.986 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f231f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f232010b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:53.986 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f231f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f232010b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:52838/0 (socket says 172.21.15.135:52838) 2023-11-15T06:38:53.986 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f231f7fe700 1 -- 172.21.15.135:0/3028186218 learned_addr learned my addr 172.21.15.135:0/3028186218 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:53.987 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f231f7fe700 1 -- 172.21.15.135:0/3028186218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2310005040 con 0x7f23200f5410 2023-11-15T06:38:53.987 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f231f7fe700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f232010b850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f2310013510 tx=0x7f231000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:53.987 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f2307fff700 1 -- 172.21.15.135:0/3028186218 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2310009320 con 0x7f23200f5410 2023-11-15T06:38:53.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2320060220 con 0x7f23200f5410 2023-11-15T06:38:53.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f2307fff700 1 -- 172.21.15.135:0/3028186218 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2310013f00 con 0x7f23200f5410 2023-11-15T06:38:53.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f2307fff700 1 -- 172.21.15.135:0/3028186218 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2310014d90 con 0x7f23200f5410 2023-11-15T06:38:53.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.980+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f232010bd40 con 0x7f23200f5410 2023-11-15T06:38:53.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.981+0000 7f2307fff700 1 -- 172.21.15.135:0/3028186218 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2310014420 con 0x7f23200f5410 2023-11-15T06:38:53.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.981+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2320045ad0 con 0x7f23200f5410 2023-11-15T06:38:53.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.981+0000 7f2307fff700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f230802ff60 0x7f2308032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:53.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.981+0000 7f2307fff700 1 -- 172.21.15.135:0/3028186218 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2310040fb0 con 0x7f23200f5410 2023-11-15T06:38:53.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.982+0000 7f231effd700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f230802ff60 0x7f2308032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:53.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.982+0000 7f231effd700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f230802ff60 0x7f2308032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f23140096a0 tx=0x7f2314006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:53.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:53.986+0000 7f2307fff700 1 -- 172.21.15.135:0/3028186218 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f23100095f0 con 0x7f23200f5410 2023-11-15T06:38:54.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:52.240933+0000 mgr.smithi123.razjtn (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:54.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.194+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f23200fae60 con 0x7f23200f5410 2023-11-15T06:38:54.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.195+0000 7f2307fff700 1 -- 172.21.15.135:0/3028186218 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f231001db70 con 0x7f23200f5410 2023-11-15T06:38:54.198 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:54.198 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:54.200 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f230802ff60 msgr2=0x7f2308032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:54.200 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f230802ff60 0x7f2308032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f23140096a0 tx=0x7f2314006b40).stop 2023-11-15T06:38:54.200 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 msgr2=0x7f232010b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:54.200 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f232010b850 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f2310013510 tx=0x7f231000b040).stop 2023-11-15T06:38:54.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 shutdown_connections 2023-11-15T06:38:54.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f230802ff60 0x7f2308032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:54.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 --2- 172.21.15.135:0/3028186218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f23200f5410 0x7f232010b850 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:54.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 >> 172.21.15.135:0/3028186218 conn(0x7f23200f0b70 msgr2=0x7f23200f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:54.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 shutdown_connections 2023-11-15T06:38:54.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:54.197+0000 7f2325b33700 1 -- 172.21.15.135:0/3028186218 wait complete. 2023-11-15T06:38:54.202 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:55.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:54 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:54.196292+0000 mon.smithi123 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.135:0/3028186218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:55.641 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:55.642 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:56.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:54.241238+0000 mgr.smithi123.razjtn (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:56.724 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.721+0000 7f51926b0700 1 -- 172.21.15.135:0/4236953563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 msgr2=0x7f518c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:56.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.721+0000 7f51926b0700 1 --2- 172.21.15.135:0/4236953563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c0f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f5180009fb0 tx=0x7f5180009b30).stop 2023-11-15T06:38:56.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.721+0000 7f51926b0700 1 -- 172.21.15.135:0/4236953563 shutdown_connections 2023-11-15T06:38:56.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.721+0000 7f51926b0700 1 --2- 172.21.15.135:0/4236953563 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c0f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:56.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.721+0000 7f51926b0700 1 -- 172.21.15.135:0/4236953563 >> 172.21.15.135:0/4236953563 conn(0x7f518c0f0b70 msgr2=0x7f518c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:56.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.721+0000 7f51926b0700 1 -- 172.21.15.135:0/4236953563 shutdown_connections 2023-11-15T06:38:56.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.721+0000 7f51926b0700 1 -- 172.21.15.135:0/4236953563 wait complete. 2023-11-15T06:38:56.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.722+0000 7f51926b0700 1 Processor -- start 2023-11-15T06:38:56.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.722+0000 7f51926b0700 1 -- start start 2023-11-15T06:38:56.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.722+0000 7f51926b0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:56.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.722+0000 7f51926b0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f518c105b20 con 0x7f518c0f5410 2023-11-15T06:38:56.729 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.722+0000 7f518bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:56.729 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.722+0000 7f518bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:52858/0 (socket says 172.21.15.135:52858) 2023-11-15T06:38:56.729 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.722+0000 7f518bfff700 1 -- 172.21.15.135:0/880361858 learned_addr learned my addr 172.21.15.135:0/880361858 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:56.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.723+0000 7f518bfff700 1 -- 172.21.15.135:0/880361858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5180005040 con 0x7f518c0f5410 2023-11-15T06:38:56.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.723+0000 7f518bfff700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c105630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5180013510 tx=0x7f518000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:56.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.723+0000 7f5188ff9700 1 -- 172.21.15.135:0/880361858 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5180009320 con 0x7f518c0f5410 2023-11-15T06:38:56.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.723+0000 7f5188ff9700 1 -- 172.21.15.135:0/880361858 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5180013f00 con 0x7f518c0f5410 2023-11-15T06:38:56.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.723+0000 7f5188ff9700 1 -- 172.21.15.135:0/880361858 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5180014d90 con 0x7f518c0f5410 2023-11-15T06:38:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.723+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f518c105d20 con 0x7f518c0f5410 2023-11-15T06:38:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.723+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f518c1061b0 con 0x7f518c0f5410 2023-11-15T06:38:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.724+0000 7f5188ff9700 1 -- 172.21.15.135:0/880361858 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f51800095f0 con 0x7f518c0f5410 2023-11-15T06:38:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.724+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f518c045ad0 con 0x7f518c0f5410 2023-11-15T06:38:56.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.724+0000 7f5188ff9700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f517402ff60 0x7f5174032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:56.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.724+0000 7f5188ff9700 1 -- 172.21.15.135:0/880361858 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5180041030 con 0x7f518c0f5410 2023-11-15T06:38:56.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.724+0000 7f518b7fe700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f517402ff60 0x7f5174032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:56.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.725+0000 7f518b7fe700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f517402ff60 0x7f5174032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f517c0096a0 tx=0x7f517c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:56.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.729+0000 7f5188ff9700 1 -- 172.21.15.135:0/880361858 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5180014420 con 0x7f518c0f5410 2023-11-15T06:38:56.940 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.938+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f518c0fae60 con 0x7f518c0f5410 2023-11-15T06:38:56.941 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.938+0000 7f5188ff9700 1 -- 172.21.15.135:0/880361858 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5180010230 con 0x7f518c0f5410 2023-11-15T06:38:56.941 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:56.941 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:56.942 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f517402ff60 msgr2=0x7f5174032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:56.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f517402ff60 0x7f5174032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7f517c0096a0 tx=0x7f517c006b40).stop 2023-11-15T06:38:56.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 msgr2=0x7f518c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:56.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c105630 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7f5180013510 tx=0x7f518000b040).stop 2023-11-15T06:38:56.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 shutdown_connections 2023-11-15T06:38:56.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f517402ff60 0x7f5174032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:56.944 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 --2- 172.21.15.135:0/880361858 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f518c0f5410 0x7f518c105630 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:56.944 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 >> 172.21.15.135:0/880361858 conn(0x7f518c0f0b70 msgr2=0x7f518c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:56.944 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 shutdown_connections 2023-11-15T06:38:56.944 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:56.940+0000 7f51926b0700 1 -- 172.21.15.135:0/880361858 wait complete. 2023-11-15T06:38:56.945 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:38:58.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:56.241483+0000 mgr.smithi123.razjtn (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:38:58.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:56.939242+0000 mon.smithi123 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.135:0/880361858' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:38:58.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:38:58.328 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:38:59.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.420+0000 7fe4db705700 1 -- 172.21.15.135:0/3820669601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 msgr2=0x7fe4d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:59.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.420+0000 7fe4db705700 1 --2- 172.21.15.135:0/3820669601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d40f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7fe4c4009fb0 tx=0x7fe4c4009b30).stop 2023-11-15T06:38:59.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.420+0000 7fe4db705700 1 -- 172.21.15.135:0/3820669601 shutdown_connections 2023-11-15T06:38:59.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.420+0000 7fe4db705700 1 --2- 172.21.15.135:0/3820669601 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d40f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:59.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.420+0000 7fe4db705700 1 -- 172.21.15.135:0/3820669601 >> 172.21.15.135:0/3820669601 conn(0x7fe4d40f0b70 msgr2=0x7fe4d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:59.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.420+0000 7fe4db705700 1 -- 172.21.15.135:0/3820669601 shutdown_connections 2023-11-15T06:38:59.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.420+0000 7fe4db705700 1 -- 172.21.15.135:0/3820669601 wait complete. 2023-11-15T06:38:59.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.421+0000 7fe4db705700 1 Processor -- start 2023-11-15T06:38:59.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.421+0000 7fe4db705700 1 -- start start 2023-11-15T06:38:59.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.421+0000 7fe4db705700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:59.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.421+0000 7fe4db705700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe4d4105b20 con 0x7fe4d40f5410 2023-11-15T06:38:59.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.421+0000 7fe4d94a1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:59.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.421+0000 7fe4d94a1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43862/0 (socket says 172.21.15.135:43862) 2023-11-15T06:38:59.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.421+0000 7fe4d94a1700 1 -- 172.21.15.135:0/3283470799 learned_addr learned my addr 172.21.15.135:0/3283470799 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:38:59.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.422+0000 7fe4d94a1700 1 -- 172.21.15.135:0/3283470799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe4c4005040 con 0x7fe4d40f5410 2023-11-15T06:38:59.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.422+0000 7fe4d94a1700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d4105630 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe4c4013510 tx=0x7fe4c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:59.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.422+0000 7fe4c9ffb700 1 -- 172.21.15.135:0/3283470799 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c4009320 con 0x7fe4d40f5410 2023-11-15T06:38:59.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.422+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe4d4105d20 con 0x7fe4d40f5410 2023-11-15T06:38:59.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.422+0000 7fe4c9ffb700 1 -- 172.21.15.135:0/3283470799 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe4c4013f00 con 0x7fe4d40f5410 2023-11-15T06:38:59.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.422+0000 7fe4c9ffb700 1 -- 172.21.15.135:0/3283470799 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe4c4014d90 con 0x7fe4d40f5410 2023-11-15T06:38:59.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.422+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe4d41061b0 con 0x7fe4d40f5410 2023-11-15T06:38:59.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.423+0000 7fe4c9ffb700 1 -- 172.21.15.135:0/3283470799 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4c40095f0 con 0x7fe4d40f5410 2023-11-15T06:38:59.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.423+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe4d4045ad0 con 0x7fe4d40f5410 2023-11-15T06:38:59.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.423+0000 7fe4c9ffb700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe4c0030000 0x7fe4c00324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:38:59.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.423+0000 7fe4c9ffb700 1 -- 172.21.15.135:0/3283470799 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4c4041040 con 0x7fe4d40f5410 2023-11-15T06:38:59.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.424+0000 7fe4d8ca0700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe4c0030000 0x7fe4c00324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:38:59.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.424+0000 7fe4d8ca0700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe4c0030000 0x7fe4c00324c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe4d00096a0 tx=0x7fe4d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:38:59.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.428+0000 7fe4c9ffb700 1 -- 172.21.15.135:0/3283470799 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe4c4014420 con 0x7fe4d40f5410 2023-11-15T06:38:59.635 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.633+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4d40fae60 con 0x7fe4d40f5410 2023-11-15T06:38:59.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.633+0000 7fe4c9ffb700 1 -- 172.21.15.135:0/3283470799 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe4c401d7f0 con 0x7fe4d40f5410 2023-11-15T06:38:59.636 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:38:59.637 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:38:59.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe4c0030000 msgr2=0x7fe4c00324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:59.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe4c0030000 0x7fe4c00324c0 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7fe4d00096a0 tx=0x7fe4d0006b40).stop 2023-11-15T06:38:59.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 msgr2=0x7fe4d4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:38:59.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d4105630 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7fe4c4013510 tx=0x7fe4c400b040).stop 2023-11-15T06:38:59.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 shutdown_connections 2023-11-15T06:38:59.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe4c0030000 0x7fe4c00324c0 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:59.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 --2- 172.21.15.135:0/3283470799 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4d40f5410 0x7fe4d4105630 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:38:59.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 >> 172.21.15.135:0/3283470799 conn(0x7fe4d40f0b70 msgr2=0x7fe4d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:38:59.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 shutdown_connections 2023-11-15T06:38:59.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:38:59.635+0000 7fe4db705700 1 -- 172.21.15.135:0/3283470799 wait complete. 2023-11-15T06:38:59.640 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:00.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:38:58.241730+0000 mgr.smithi123.razjtn (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:00.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:38:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:38:59.634386+0000 mon.smithi123 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.135:0/3283470799' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:01.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:01.014 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:02.103 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 -- 172.21.15.135:0/3780919379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 msgr2=0x7f92d00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:02.103 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 --2- 172.21.15.135:0/3780919379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d00f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f92b8009fb0 tx=0x7f92b8009b30).stop 2023-11-15T06:39:02.104 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 -- 172.21.15.135:0/3780919379 shutdown_connections 2023-11-15T06:39:02.104 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 --2- 172.21.15.135:0/3780919379 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d00f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:02.104 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 -- 172.21.15.135:0/3780919379 >> 172.21.15.135:0/3780919379 conn(0x7f92d00f1c70 msgr2=0x7f92d00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:02.104 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 -- 172.21.15.135:0/3780919379 shutdown_connections 2023-11-15T06:39:02.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 -- 172.21.15.135:0/3780919379 wait complete. 2023-11-15T06:39:02.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.099+0000 7f92d5bc7700 1 Processor -- start 2023-11-15T06:39:02.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.100+0000 7f92d5bc7700 1 -- start start 2023-11-15T06:39:02.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.100+0000 7f92d5bc7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:02.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.100+0000 7f92d5bc7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f92d0106c20 con 0x7f92d00f6510 2023-11-15T06:39:02.106 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.100+0000 7f92cf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:02.106 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.100+0000 7f92cf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43870/0 (socket says 172.21.15.135:43870) 2023-11-15T06:39:02.106 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.100+0000 7f92cf7fe700 1 -- 172.21.15.135:0/937281813 learned_addr learned my addr 172.21.15.135:0/937281813 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:02.106 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.100+0000 7f92cf7fe700 1 -- 172.21.15.135:0/937281813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f92b8005040 con 0x7f92d00f6510 2023-11-15T06:39:02.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.101+0000 7f92cf7fe700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d0106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f92b8013510 tx=0x7f92b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:02.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.101+0000 7f92c7fff700 1 -- 172.21.15.135:0/937281813 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b8009320 con 0x7f92d00f6510 2023-11-15T06:39:02.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.101+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f92d0106e20 con 0x7f92d00f6510 2023-11-15T06:39:02.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.101+0000 7f92c7fff700 1 -- 172.21.15.135:0/937281813 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f92b8013f00 con 0x7f92d00f6510 2023-11-15T06:39:02.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.101+0000 7f92c7fff700 1 -- 172.21.15.135:0/937281813 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f92b8014d90 con 0x7f92d00f6510 2023-11-15T06:39:02.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.101+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f92d01072b0 con 0x7f92d00f6510 2023-11-15T06:39:02.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.102+0000 7f92c7fff700 1 -- 172.21.15.135:0/937281813 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f92b80263a0 con 0x7f92d00f6510 2023-11-15T06:39:02.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.102+0000 7f92c7fff700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f92bc02ff60 0x7f92bc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:02.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.102+0000 7f92c7fff700 1 -- 172.21.15.135:0/937281813 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f92b8041000 con 0x7f92d00f6510 2023-11-15T06:39:02.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.102+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f92d0045ad0 con 0x7f92d00f6510 2023-11-15T06:39:02.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.102+0000 7f92ceffd700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f92bc02ff60 0x7f92bc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:02.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.102+0000 7f92ceffd700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f92bc02ff60 0x7f92bc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f92c00096a0 tx=0x7f92c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:02.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.106+0000 7f92c7fff700 1 -- 172.21.15.135:0/937281813 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f92b8012920 con 0x7f92d00f6510 2023-11-15T06:39:02.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:00.241981+0000 mgr.smithi123.razjtn (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:02.314 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.311+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f92d00fc050 con 0x7f92d00f6510 2023-11-15T06:39:02.315 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.312+0000 7f92c7fff700 1 -- 172.21.15.135:0/937281813 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f92b802b020 con 0x7f92d00f6510 2023-11-15T06:39:02.315 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:02.315 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:02.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f92bc02ff60 msgr2=0x7f92bc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:02.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f92bc02ff60 0x7f92bc032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7f92c00096a0 tx=0x7f92c0006b40).stop 2023-11-15T06:39:02.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 msgr2=0x7f92d0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:02.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d0106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f92b8013510 tx=0x7f92b800b040).stop 2023-11-15T06:39:02.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 shutdown_connections 2023-11-15T06:39:02.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f92bc02ff60 0x7f92bc032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:02.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 --2- 172.21.15.135:0/937281813 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f92d00f6510 0x7f92d0106730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:02.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 >> 172.21.15.135:0/937281813 conn(0x7f92d00f1c70 msgr2=0x7f92d00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:02.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 shutdown_connections 2023-11-15T06:39:02.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:02.314+0000 7f92d5bc7700 1 -- 172.21.15.135:0/937281813 wait complete. 2023-11-15T06:39:02.318 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:03.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:02 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:02.313111+0000 mon.smithi123 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.135:0/937281813' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:03.692 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:03.693 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:04.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:02.242256+0000 mgr.smithi123.razjtn (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:04.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.840+0000 7f13806bd700 1 -- 172.21.15.135:0/2940740018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 msgr2=0x7f13780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:04.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.840+0000 7f13806bd700 1 --2- 172.21.15.135:0/2940740018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f13780f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f1368009fb0 tx=0x7f1368009b30).stop 2023-11-15T06:39:04.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.840+0000 7f13806bd700 1 -- 172.21.15.135:0/2940740018 shutdown_connections 2023-11-15T06:39:04.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.840+0000 7f13806bd700 1 --2- 172.21.15.135:0/2940740018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f13780f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:04.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.840+0000 7f13806bd700 1 -- 172.21.15.135:0/2940740018 >> 172.21.15.135:0/2940740018 conn(0x7f13780f0b70 msgr2=0x7f13780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:04.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.840+0000 7f13806bd700 1 -- 172.21.15.135:0/2940740018 shutdown_connections 2023-11-15T06:39:04.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.840+0000 7f13806bd700 1 -- 172.21.15.135:0/2940740018 wait complete. 2023-11-15T06:39:04.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.841+0000 7f13806bd700 1 Processor -- start 2023-11-15T06:39:04.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.841+0000 7f13806bd700 1 -- start start 2023-11-15T06:39:04.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.841+0000 7f13806bd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f1378109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:04.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.841+0000 7f13806bd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1378060240 con 0x7f13780f5410 2023-11-15T06:39:04.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.841+0000 7f137e459700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f1378109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:04.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.841+0000 7f137e459700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f1378109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43882/0 (socket says 172.21.15.135:43882) 2023-11-15T06:39:04.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.841+0000 7f137e459700 1 -- 172.21.15.135:0/3132401215 learned_addr learned my addr 172.21.15.135:0/3132401215 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:04.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.842+0000 7f137e459700 1 -- 172.21.15.135:0/3132401215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1368005040 con 0x7f13780f5410 2023-11-15T06:39:04.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.842+0000 7f137e459700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f1378109430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f1368013510 tx=0x7f136800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:04.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.842+0000 7f136effd700 1 -- 172.21.15.135:0/3132401215 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1368009320 con 0x7f13780f5410 2023-11-15T06:39:04.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.842+0000 7f136effd700 1 -- 172.21.15.135:0/3132401215 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1368013f00 con 0x7f13780f5410 2023-11-15T06:39:04.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.842+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1378060440 con 0x7f13780f5410 2023-11-15T06:39:04.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.842+0000 7f136effd700 1 -- 172.21.15.135:0/3132401215 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1368014d90 con 0x7f13780f5410 2023-11-15T06:39:04.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.842+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1378109ae0 con 0x7f13780f5410 2023-11-15T06:39:04.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.843+0000 7f136effd700 1 -- 172.21.15.135:0/3132401215 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13680095f0 con 0x7f13780f5410 2023-11-15T06:39:04.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.843+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1378045ad0 con 0x7f13780f5410 2023-11-15T06:39:04.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.843+0000 7f136effd700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f136402ff60 0x7f1364032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:04.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.843+0000 7f136effd700 1 -- 172.21.15.135:0/3132401215 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1368040fe0 con 0x7f13780f5410 2023-11-15T06:39:04.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.843+0000 7f137dc58700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f136402ff60 0x7f1364032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:04.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.843+0000 7f137dc58700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f136402ff60 0x7f1364032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f13740096a0 tx=0x7f1374006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:04.851 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:04.847+0000 7f136effd700 1 -- 172.21.15.135:0/3132401215 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f136801d7f0 con 0x7f13780f5410 2023-11-15T06:39:05.058 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.056+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13780fae60 con 0x7f13780f5410 2023-11-15T06:39:05.059 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.056+0000 7f136effd700 1 -- 172.21.15.135:0/3132401215 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1368010230 con 0x7f13780f5410 2023-11-15T06:39:05.059 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:05.060 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:05.061 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f136402ff60 msgr2=0x7f1364032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:05.061 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f136402ff60 0x7f1364032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f13740096a0 tx=0x7f1374006b40).stop 2023-11-15T06:39:05.061 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 msgr2=0x7f1378109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:05.061 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f1378109430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f1368013510 tx=0x7f136800b040).stop 2023-11-15T06:39:05.061 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 shutdown_connections 2023-11-15T06:39:05.062 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f136402ff60 0x7f1364032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:05.062 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 --2- 172.21.15.135:0/3132401215 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f13780f5410 0x7f1378109430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:05.062 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 >> 172.21.15.135:0/3132401215 conn(0x7f13780f0b70 msgr2=0x7f13780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:05.062 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 shutdown_connections 2023-11-15T06:39:05.063 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:05.058+0000 7f13806bd700 1 -- 172.21.15.135:0/3132401215 wait complete. 2023-11-15T06:39:05.063 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:06.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:04.242597+0000 mgr.smithi123.razjtn (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:06.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:05.057329+0000 mon.smithi123 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.135:0/3132401215' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:06.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:06.439 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:07.606 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.602+0000 7f0a4a46c700 1 -- 172.21.15.135:0/853368821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 msgr2=0x7f0a440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:07.606 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.602+0000 7f0a4a46c700 1 --2- 172.21.15.135:0/853368821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a440f66c0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7f0a40009fb0 tx=0x7f0a40009b30).stop 2023-11-15T06:39:07.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.602+0000 7f0a4a46c700 1 -- 172.21.15.135:0/853368821 shutdown_connections 2023-11-15T06:39:07.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.602+0000 7f0a4a46c700 1 --2- 172.21.15.135:0/853368821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a440f66c0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:07.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.602+0000 7f0a4a46c700 1 -- 172.21.15.135:0/853368821 >> 172.21.15.135:0/853368821 conn(0x7f0a440f1aa0 msgr2=0x7f0a440f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:07.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.603+0000 7f0a4a46c700 1 -- 172.21.15.135:0/853368821 shutdown_connections 2023-11-15T06:39:07.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.603+0000 7f0a4a46c700 1 -- 172.21.15.135:0/853368821 wait complete. 2023-11-15T06:39:07.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.603+0000 7f0a4a46c700 1 Processor -- start 2023-11-15T06:39:07.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.603+0000 7f0a4a46c700 1 -- start start 2023-11-15T06:39:07.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.604+0000 7f0a4a46c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a441064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:07.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.604+0000 7f0a4a46c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a441069a0 con 0x7f0a440f62a0 2023-11-15T06:39:07.610 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.604+0000 7f0a4946a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a441064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:07.610 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.604+0000 7f0a4946a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a441064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43902/0 (socket says 172.21.15.135:43902) 2023-11-15T06:39:07.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.604+0000 7f0a4946a700 1 -- 172.21.15.135:0/305914933 learned_addr learned my addr 172.21.15.135:0/305914933 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:07.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.604+0000 7f0a4946a700 1 -- 172.21.15.135:0/305914933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a40005040 con 0x7f0a440f62a0 2023-11-15T06:39:07.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.604+0000 7f0a4946a700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a441064b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0a40013510 tx=0x7f0a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:07.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.605+0000 7f0a39ffb700 1 -- 172.21.15.135:0/305914933 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a40009320 con 0x7f0a440f62a0 2023-11-15T06:39:07.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.605+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a44106ba0 con 0x7f0a440f62a0 2023-11-15T06:39:07.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.605+0000 7f0a39ffb700 1 -- 172.21.15.135:0/305914933 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a40013f00 con 0x7f0a440f62a0 2023-11-15T06:39:07.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.605+0000 7f0a39ffb700 1 -- 172.21.15.135:0/305914933 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a40014dc0 con 0x7f0a440f62a0 2023-11-15T06:39:07.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.605+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a44107030 con 0x7f0a440f62a0 2023-11-15T06:39:07.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.606+0000 7f0a39ffb700 1 -- 172.21.15.135:0/305914933 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a400095f0 con 0x7f0a440f62a0 2023-11-15T06:39:07.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.606+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a44045ad0 con 0x7f0a440f62a0 2023-11-15T06:39:07.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.606+0000 7f0a39ffb700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0a3002ff60 0x7f0a30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:07.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.606+0000 7f0a39ffb700 1 -- 172.21.15.135:0/305914933 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a40041100 con 0x7f0a440f62a0 2023-11-15T06:39:07.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.606+0000 7f0a48c69700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0a3002ff60 0x7f0a30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:07.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.606+0000 7f0a48c69700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0a3002ff60 0x7f0a30032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0a340096a0 tx=0x7f0a34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:07.614 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.610+0000 7f0a39ffb700 1 -- 172.21.15.135:0/305914933 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a40014420 con 0x7f0a440f62a0 2023-11-15T06:39:07.821 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.818+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a44107ee0 con 0x7f0a440f62a0 2023-11-15T06:39:07.822 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.819+0000 7f0a39ffb700 1 -- 172.21.15.135:0/305914933 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0a40010230 con 0x7f0a440f62a0 2023-11-15T06:39:07.822 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:07.822 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:07.824 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0a3002ff60 msgr2=0x7f0a30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:07.824 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0a3002ff60 0x7f0a30032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f0a340096a0 tx=0x7f0a34006b40).stop 2023-11-15T06:39:07.824 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 msgr2=0x7f0a441064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:07.824 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a441064b0 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f0a40013510 tx=0x7f0a4000b040).stop 2023-11-15T06:39:07.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 shutdown_connections 2023-11-15T06:39:07.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0a3002ff60 0x7f0a30032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:07.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 --2- 172.21.15.135:0/305914933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0a440f62a0 0x7f0a441064b0 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:07.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 >> 172.21.15.135:0/305914933 conn(0x7f0a440f1aa0 msgr2=0x7f0a440f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:07.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 shutdown_connections 2023-11-15T06:39:07.826 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:07.821+0000 7f0a4a46c700 1 -- 172.21.15.135:0/305914933 wait complete. 2023-11-15T06:39:07.826 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:08.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:06.242874+0000 mgr.smithi123.razjtn (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:08.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:07.820259+0000 mon.smithi123 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.135:0/305914933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:09.205 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:09.206 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:10.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:08.243145+0000 mgr.smithi123.razjtn (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:10.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 -- 172.21.15.135:0/3327162260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 msgr2=0x7f076c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:10.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 --2- 172.21.15.135:0/3327162260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c0f55c0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f075c009fb0 tx=0x7f075c009b30).stop 2023-11-15T06:39:10.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 -- 172.21.15.135:0/3327162260 shutdown_connections 2023-11-15T06:39:10.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 --2- 172.21.15.135:0/3327162260 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c0f55c0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:10.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 -- 172.21.15.135:0/3327162260 >> 172.21.15.135:0/3327162260 conn(0x7f076c0f09a0 msgr2=0x7f076c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:10.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 -- 172.21.15.135:0/3327162260 shutdown_connections 2023-11-15T06:39:10.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 -- 172.21.15.135:0/3327162260 wait complete. 2023-11-15T06:39:10.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 Processor -- start 2023-11-15T06:39:10.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 -- start start 2023-11-15T06:39:10.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:10.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.303+0000 7f077213c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f076c1058a0 con 0x7f076c0f51a0 2023-11-15T06:39:10.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f077113a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:10.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f077113a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43732/0 (socket says 172.21.15.135:43732) 2023-11-15T06:39:10.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f077113a700 1 -- 172.21.15.135:0/1898333469 learned_addr learned my addr 172.21.15.135:0/1898333469 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:10.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f077113a700 1 -- 172.21.15.135:0/1898333469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f075c005040 con 0x7f076c0f51a0 2023-11-15T06:39:10.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f077113a700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c1053b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f075c013510 tx=0x7f075c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:10.311 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f0761ffb700 1 -- 172.21.15.135:0/1898333469 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f075c009320 con 0x7f076c0f51a0 2023-11-15T06:39:10.311 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f076c105aa0 con 0x7f076c0f51a0 2023-11-15T06:39:10.311 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f0761ffb700 1 -- 172.21.15.135:0/1898333469 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f075c013f00 con 0x7f076c0f51a0 2023-11-15T06:39:10.311 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f0761ffb700 1 -- 172.21.15.135:0/1898333469 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f075c014dc0 con 0x7f076c0f51a0 2023-11-15T06:39:10.312 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.304+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f076c105f30 con 0x7f076c0f51a0 2023-11-15T06:39:10.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.305+0000 7f0761ffb700 1 -- 172.21.15.135:0/1898333469 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f075c01d3d0 con 0x7f076c0f51a0 2023-11-15T06:39:10.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.305+0000 7f0761ffb700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f075802ff60 0x7f0758032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:10.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.305+0000 7f0761ffb700 1 -- 172.21.15.135:0/1898333469 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f075c041100 con 0x7f076c0f51a0 2023-11-15T06:39:10.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.305+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f076c045ad0 con 0x7f076c0f51a0 2023-11-15T06:39:10.314 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.305+0000 7f0770939700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f075802ff60 0x7f0758032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:10.314 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.306+0000 7f0770939700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f075802ff60 0x7f0758032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f07680096a0 tx=0x7f0768006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:10.314 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.310+0000 7f0761ffb700 1 -- 172.21.15.135:0/1898333469 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f075c012920 con 0x7f076c0f51a0 2023-11-15T06:39:10.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.519+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f076c0f1b70 con 0x7f076c0f51a0 2023-11-15T06:39:10.523 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.520+0000 7f0761ffb700 1 -- 172.21.15.135:0/1898333469 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f075c012330 con 0x7f076c0f51a0 2023-11-15T06:39:10.523 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:10.524 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:10.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f075802ff60 msgr2=0x7f0758032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:10.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f075802ff60 0x7f0758032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f07680096a0 tx=0x7f0768006b40).stop 2023-11-15T06:39:10.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 msgr2=0x7f076c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:10.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c1053b0 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f075c013510 tx=0x7f075c00b040).stop 2023-11-15T06:39:10.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 shutdown_connections 2023-11-15T06:39:10.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f075802ff60 0x7f0758032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:10.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 --2- 172.21.15.135:0/1898333469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f076c0f51a0 0x7f076c1053b0 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:10.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 >> 172.21.15.135:0/1898333469 conn(0x7f076c0f09a0 msgr2=0x7f076c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:10.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 shutdown_connections 2023-11-15T06:39:10.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:10.522+0000 7f077213c700 1 -- 172.21.15.135:0/1898333469 wait complete. 2023-11-15T06:39:10.527 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:11.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:10 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:10.521252+0000 mon.smithi123 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.135:0/1898333469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:11.896 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:11.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:12.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:10.243428+0000 mgr.smithi123.razjtn (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:13.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 -- 172.21.15.135:0/4269461346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 msgr2=0x7fd9f80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:13.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 --2- 172.21.15.135:0/4269461346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f80f66c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fd9e8009fb0 tx=0x7fd9e8009b30).stop 2023-11-15T06:39:13.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 -- 172.21.15.135:0/4269461346 shutdown_connections 2023-11-15T06:39:13.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 --2- 172.21.15.135:0/4269461346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f80f66c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:13.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 -- 172.21.15.135:0/4269461346 >> 172.21.15.135:0/4269461346 conn(0x7fd9f80f1aa0 msgr2=0x7fd9f80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:13.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 -- 172.21.15.135:0/4269461346 shutdown_connections 2023-11-15T06:39:13.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 -- 172.21.15.135:0/4269461346 wait complete. 2023-11-15T06:39:13.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 Processor -- start 2023-11-15T06:39:13.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.002+0000 7fd9fd59d700 1 -- start start 2023-11-15T06:39:13.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.003+0000 7fd9fd59d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:13.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.003+0000 7fd9fd59d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd9f8069e20 con 0x7fd9f80f62a0 2023-11-15T06:39:13.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.003+0000 7fd9f7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:13.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.003+0000 7fd9f7fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43750/0 (socket says 172.21.15.135:43750) 2023-11-15T06:39:13.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.003+0000 7fd9f7fff700 1 -- 172.21.15.135:0/1357020032 learned_addr learned my addr 172.21.15.135:0/1357020032 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:13.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.003+0000 7fd9f7fff700 1 -- 172.21.15.135:0/1357020032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd9e8005040 con 0x7fd9f80f62a0 2023-11-15T06:39:13.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.004+0000 7fd9f7fff700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f806b9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fd9e8013510 tx=0x7fd9e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:13.013 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.004+0000 7fd9f4ff9700 1 -- 172.21.15.135:0/1357020032 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9e8009320 con 0x7fd9f80f62a0 2023-11-15T06:39:13.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.004+0000 7fd9f4ff9700 1 -- 172.21.15.135:0/1357020032 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd9e8013f00 con 0x7fd9f80f62a0 2023-11-15T06:39:13.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.004+0000 7fd9f4ff9700 1 -- 172.21.15.135:0/1357020032 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd9e8014dc0 con 0x7fd9f80f62a0 2023-11-15T06:39:13.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.004+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd9f8069fc0 con 0x7fd9f80f62a0 2023-11-15T06:39:13.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.004+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd9f806a450 con 0x7fd9f80f62a0 2023-11-15T06:39:13.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.005+0000 7fd9f4ff9700 1 -- 172.21.15.135:0/1357020032 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd9e80095f0 con 0x7fd9f80f62a0 2023-11-15T06:39:13.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.005+0000 7fd9f4ff9700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd9e0030000 0x7fd9e00324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:13.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.005+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd9f8059100 con 0x7fd9f80f62a0 2023-11-15T06:39:13.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.005+0000 7fd9f4ff9700 1 -- 172.21.15.135:0/1357020032 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd9e80410b0 con 0x7fd9f80f62a0 2023-11-15T06:39:13.017 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.006+0000 7fd9f77fe700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd9e0030000 0x7fd9e00324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:13.017 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.006+0000 7fd9f77fe700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd9e0030000 0x7fd9e00324c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fd9ec0096a0 tx=0x7fd9ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:13.017 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.010+0000 7fd9f4ff9700 1 -- 172.21.15.135:0/1357020032 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd9e8014420 con 0x7fd9f80f62a0 2023-11-15T06:39:13.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.219+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd9f80fbce0 con 0x7fd9f80f62a0 2023-11-15T06:39:13.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.221+0000 7fd9f4ff9700 1 -- 172.21.15.135:0/1357020032 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd9e802c090 con 0x7fd9f80f62a0 2023-11-15T06:39:13.224 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:13.224 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:13.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd9e0030000 msgr2=0x7fd9e00324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:13.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd9e0030000 0x7fd9e00324c0 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7fd9ec0096a0 tx=0x7fd9ec006b40).stop 2023-11-15T06:39:13.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 msgr2=0x7fd9f806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:13.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f806b9a0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7fd9e8013510 tx=0x7fd9e800b040).stop 2023-11-15T06:39:13.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 shutdown_connections 2023-11-15T06:39:13.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd9e0030000 0x7fd9e00324c0 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:13.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 --2- 172.21.15.135:0/1357020032 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd9f80f62a0 0x7fd9f806b9a0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:13.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.222+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 >> 172.21.15.135:0/1357020032 conn(0x7fd9f80f1aa0 msgr2=0x7fd9f80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:13.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.223+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 shutdown_connections 2023-11-15T06:39:13.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:13.223+0000 7fd9fd59d700 1 -- 172.21.15.135:0/1357020032 wait complete. 2023-11-15T06:39:13.228 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:14.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:12.243798+0000 mgr.smithi123.razjtn (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:14.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:13.221841+0000 mon.smithi123 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.135:0/1357020032' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:14.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:14.596 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:15.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.821+0000 7f5fe38b3700 1 -- 172.21.15.135:0/1594494299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 msgr2=0x7f5fdc0f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:15.826 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.821+0000 7f5fe38b3700 1 --2- 172.21.15.135:0/1594494299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc0f9150 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f5fcc009fb0 tx=0x7f5fcc009b30).stop 2023-11-15T06:39:15.826 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.821+0000 7f5fe38b3700 1 -- 172.21.15.135:0/1594494299 shutdown_connections 2023-11-15T06:39:15.826 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.821+0000 7f5fe38b3700 1 --2- 172.21.15.135:0/1594494299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc0f9150 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:15.826 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.821+0000 7f5fe38b3700 1 -- 172.21.15.135:0/1594494299 >> 172.21.15.135:0/1594494299 conn(0x7f5fdc0f0b70 msgr2=0x7f5fdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:15.827 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.821+0000 7f5fe38b3700 1 -- 172.21.15.135:0/1594494299 shutdown_connections 2023-11-15T06:39:15.827 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.821+0000 7f5fe38b3700 1 -- 172.21.15.135:0/1594494299 wait complete. 2023-11-15T06:39:15.827 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.822+0000 7f5fe38b3700 1 Processor -- start 2023-11-15T06:39:15.827 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.822+0000 7f5fe38b3700 1 -- start start 2023-11-15T06:39:15.827 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.822+0000 7f5fe38b3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:15.828 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.822+0000 7f5fe38b3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5fdc106bd0 con 0x7f5fdc060220 2023-11-15T06:39:15.828 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.822+0000 7f5fe164f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:15.828 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.822+0000 7f5fe164f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43764/0 (socket says 172.21.15.135:43764) 2023-11-15T06:39:15.828 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.822+0000 7f5fe164f700 1 -- 172.21.15.135:0/2100207714 learned_addr learned my addr 172.21.15.135:0/2100207714 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:15.829 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.823+0000 7f5fe164f700 1 -- 172.21.15.135:0/2100207714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5fcc005040 con 0x7f5fdc060220 2023-11-15T06:39:15.829 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.823+0000 7f5fe164f700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc1066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5fcc009830 tx=0x7f5fcc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:15.829 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.823+0000 7f5fd1ffb700 1 -- 172.21.15.135:0/2100207714 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fcc009320 con 0x7f5fdc060220 2023-11-15T06:39:15.829 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.823+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5fdc106dd0 con 0x7f5fdc060220 2023-11-15T06:39:15.829 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.823+0000 7f5fd1ffb700 1 -- 172.21.15.135:0/2100207714 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5fcc0136c0 con 0x7f5fdc060220 2023-11-15T06:39:15.830 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.823+0000 7f5fd1ffb700 1 -- 172.21.15.135:0/2100207714 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5fcc01ee90 con 0x7f5fdc060220 2023-11-15T06:39:15.830 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.823+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5fdc107260 con 0x7f5fdc060220 2023-11-15T06:39:15.831 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.824+0000 7f5fd1ffb700 1 -- 172.21.15.135:0/2100207714 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5fcc014420 con 0x7f5fdc060220 2023-11-15T06:39:15.831 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.824+0000 7f5fd1ffb700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5fc802ff10 0x7f5fc80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:15.831 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.824+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5fdc045ad0 con 0x7f5fdc060220 2023-11-15T06:39:15.832 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.824+0000 7f5fd1ffb700 1 -- 172.21.15.135:0/2100207714 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5fcc040fd0 con 0x7f5fdc060220 2023-11-15T06:39:15.832 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.824+0000 7f5fe0e4e700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5fc802ff10 0x7f5fc80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:15.832 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.825+0000 7f5fe0e4e700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5fc802ff10 0x7f5fc80323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5fd80096a0 tx=0x7f5fd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:15.832 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:15.829+0000 7f5fd1ffb700 1 -- 172.21.15.135:0/2100207714 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5fcc0273f0 con 0x7f5fdc060220 2023-11-15T06:39:16.036 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.032+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5fdc108110 con 0x7f5fdc060220 2023-11-15T06:39:16.036 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.033+0000 7f5fd1ffb700 1 -- 172.21.15.135:0/2100207714 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5fcc01c320 con 0x7f5fdc060220 2023-11-15T06:39:16.037 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:16.037 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:16.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5fc802ff10 msgr2=0x7f5fc80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:16.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5fc802ff10 0x7f5fc80323d0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f5fd80096a0 tx=0x7f5fd8006b40).stop 2023-11-15T06:39:16.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 msgr2=0x7f5fdc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:16.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc1066e0 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f5fcc009830 tx=0x7f5fcc00b040).stop 2023-11-15T06:39:16.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 shutdown_connections 2023-11-15T06:39:16.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5fc802ff10 0x7f5fc80323d0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:16.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 --2- 172.21.15.135:0/2100207714 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5fdc060220 0x7f5fdc1066e0 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:16.042 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 >> 172.21.15.135:0/2100207714 conn(0x7f5fdc0f0b70 msgr2=0x7f5fdc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:16.043 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 shutdown_connections 2023-11-15T06:39:16.043 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:16.034+0000 7f5fe38b3700 1 -- 172.21.15.135:0/2100207714 wait complete. 2023-11-15T06:39:16.043 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:16.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:14.244074+0000 mgr.smithi123.razjtn (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:17.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:16 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:16.033786+0000 mon.smithi123 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.135:0/2100207714' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:17.411 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:17.411 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:18.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:16.244433+0000 mgr.smithi123.razjtn (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:18.540 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.537+0000 7f6feef28700 1 -- 172.21.15.135:0/2237895364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 msgr2=0x7f6fe80f7f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:18.541 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.537+0000 7f6feef28700 1 --2- 172.21.15.135:0/2237895364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe80f7f80 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f6fdc009fb0 tx=0x7f6fdc009b30).stop 2023-11-15T06:39:18.541 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.537+0000 7f6feef28700 1 -- 172.21.15.135:0/2237895364 shutdown_connections 2023-11-15T06:39:18.541 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.537+0000 7f6feef28700 1 --2- 172.21.15.135:0/2237895364 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe80f7f80 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:18.542 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.537+0000 7f6feef28700 1 -- 172.21.15.135:0/2237895364 >> 172.21.15.135:0/2237895364 conn(0x7f6fe80f02e0 msgr2=0x7f6fe80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:18.542 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.537+0000 7f6feef28700 1 -- 172.21.15.135:0/2237895364 shutdown_connections 2023-11-15T06:39:18.542 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.537+0000 7f6feef28700 1 -- 172.21.15.135:0/2237895364 wait complete. 2023-11-15T06:39:18.545 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.538+0000 7f6feef28700 1 Processor -- start 2023-11-15T06:39:18.545 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.538+0000 7f6feef28700 1 -- start start 2023-11-15T06:39:18.546 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.538+0000 7f6feef28700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe8106740 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:18.546 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.538+0000 7f6feef28700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fe8106c30 con 0x7f6fe80f7b60 2023-11-15T06:39:18.546 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6feccc4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe8106740 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:18.546 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6feccc4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe8106740 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55876/0 (socket says 172.21.15.135:55876) 2023-11-15T06:39:18.546 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6feccc4700 1 -- 172.21.15.135:0/4242389716 learned_addr learned my addr 172.21.15.135:0/4242389716 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:18.547 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6feccc4700 1 -- 172.21.15.135:0/4242389716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6fdc005040 con 0x7f6fe80f7b60 2023-11-15T06:39:18.547 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6feccc4700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe8106740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f6fdc014000 tx=0x7f6fdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:18.547 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6fe57fa700 1 -- 172.21.15.135:0/4242389716 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fdc0095f0 con 0x7f6fe80f7b60 2023-11-15T06:39:18.547 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fe8106e30 con 0x7f6fe80f7b60 2023-11-15T06:39:18.547 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6fe57fa700 1 -- 172.21.15.135:0/4242389716 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6fdc007e70 con 0x7f6fe80f7b60 2023-11-15T06:39:18.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6fe57fa700 1 -- 172.21.15.135:0/4242389716 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6fdc014ea0 con 0x7f6fe80f7b60 2023-11-15T06:39:18.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.539+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fe81072c0 con 0x7f6fe80f7b60 2023-11-15T06:39:18.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.540+0000 7f6fe57fa700 1 -- 172.21.15.135:0/4242389716 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6fdc025070 con 0x7f6fe80f7b60 2023-11-15T06:39:18.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.540+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fe8045ad0 con 0x7f6fe80f7b60 2023-11-15T06:39:18.548 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.540+0000 7f6fe57fa700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6fd802ff60 0x7f6fd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:18.549 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.540+0000 7f6fe57fa700 1 -- 172.21.15.135:0/4242389716 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6fdc040930 con 0x7f6fe80f7b60 2023-11-15T06:39:18.549 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.540+0000 7f6fe7fff700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6fd802ff60 0x7f6fd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:18.549 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.541+0000 7f6fe7fff700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6fd802ff60 0x7f6fd8032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f6fd40096a0 tx=0x7f6fd4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:18.550 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.545+0000 7f6fe57fa700 1 -- 172.21.15.135:0/4242389716 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6fdc00f4e0 con 0x7f6fe80f7b60 2023-11-15T06:39:18.752 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.750+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fe8108170 con 0x7f6fe80f7b60 2023-11-15T06:39:18.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.750+0000 7f6fe57fa700 1 -- 172.21.15.135:0/4242389716 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6fdc00f300 con 0x7f6fe80f7b60 2023-11-15T06:39:18.753 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:18.754 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:18.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6fd802ff60 msgr2=0x7f6fd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:18.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6fd802ff60 0x7f6fd8032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f6fd40096a0 tx=0x7f6fd4006b40).stop 2023-11-15T06:39:18.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 msgr2=0x7f6fe8106740 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:18.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe8106740 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f6fdc014000 tx=0x7f6fdc00b040).stop 2023-11-15T06:39:18.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 shutdown_connections 2023-11-15T06:39:18.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6fd802ff60 0x7f6fd8032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:18.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 --2- 172.21.15.135:0/4242389716 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6fe80f7b60 0x7f6fe8106740 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:18.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 >> 172.21.15.135:0/4242389716 conn(0x7f6fe80f02e0 msgr2=0x7f6fe80f0e20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:18.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 shutdown_connections 2023-11-15T06:39:18.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:18.752+0000 7f6feef28700 1 -- 172.21.15.135:0/4242389716 wait complete. 2023-11-15T06:39:18.757 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:19.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:18 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:18.751468+0000 mon.smithi123 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.135:0/4242389716' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:20.139 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:20.140 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:20.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:18.244714+0000 mgr.smithi123.razjtn (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:21.221 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.218+0000 7f19f4bb4700 1 -- 172.21.15.135:0/1889788382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 msgr2=0x7f19f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:21.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.218+0000 7f19f4bb4700 1 --2- 172.21.15.135:0/1889788382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f00f6930 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f19e0009960 tx=0x7f19e0004d10).stop 2023-11-15T06:39:21.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.218+0000 7f19f4bb4700 1 -- 172.21.15.135:0/1889788382 shutdown_connections 2023-11-15T06:39:21.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.218+0000 7f19f4bb4700 1 --2- 172.21.15.135:0/1889788382 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f00f6930 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:21.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.218+0000 7f19f4bb4700 1 -- 172.21.15.135:0/1889788382 >> 172.21.15.135:0/1889788382 conn(0x7f19f00f1c70 msgr2=0x7f19f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:21.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.218+0000 7f19f4bb4700 1 -- 172.21.15.135:0/1889788382 shutdown_connections 2023-11-15T06:39:21.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.218+0000 7f19f4bb4700 1 -- 172.21.15.135:0/1889788382 wait complete. 2023-11-15T06:39:21.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19f4bb4700 1 Processor -- start 2023-11-15T06:39:21.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19f4bb4700 1 -- start start 2023-11-15T06:39:21.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19f4bb4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:21.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19f4bb4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19f0106c20 con 0x7f19f00f6510 2023-11-15T06:39:21.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19ee59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:21.225 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19ee59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55890/0 (socket says 172.21.15.135:55890) 2023-11-15T06:39:21.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19ee59c700 1 -- 172.21.15.135:0/3662506110 learned_addr learned my addr 172.21.15.135:0/3662506110 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:21.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.219+0000 7f19ee59c700 1 -- 172.21.15.135:0/3662506110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19e0009640 con 0x7f19f00f6510 2023-11-15T06:39:21.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.220+0000 7f19ee59c700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f0106730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f19e0004b70 tx=0x7f19e0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:21.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.220+0000 7f19e6ffd700 1 -- 172.21.15.135:0/3662506110 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19e0018070 con 0x7f19f00f6510 2023-11-15T06:39:21.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.220+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19f0106e20 con 0x7f19f00f6510 2023-11-15T06:39:21.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.220+0000 7f19e6ffd700 1 -- 172.21.15.135:0/3662506110 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19e000b870 con 0x7f19f00f6510 2023-11-15T06:39:21.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.220+0000 7f19e6ffd700 1 -- 172.21.15.135:0/3662506110 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19e001c560 con 0x7f19f00f6510 2023-11-15T06:39:21.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.220+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19f01072b0 con 0x7f19f00f6510 2023-11-15T06:39:21.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.221+0000 7f19e6ffd700 1 -- 172.21.15.135:0/3662506110 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19e0013630 con 0x7f19f00f6510 2023-11-15T06:39:21.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.221+0000 7f19e6ffd700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19dc02ff60 0x7f19dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:21.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.221+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19f0045ad0 con 0x7f19f00f6510 2023-11-15T06:39:21.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.221+0000 7f19e6ffd700 1 -- 172.21.15.135:0/3662506110 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19e00411d0 con 0x7f19f00f6510 2023-11-15T06:39:21.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.221+0000 7f19edd9b700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19dc02ff60 0x7f19dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:21.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.221+0000 7f19edd9b700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19dc02ff60 0x7f19dc032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f19d80096a0 tx=0x7f19d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:21.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.225+0000 7f19e6ffd700 1 -- 172.21.15.135:0/3662506110 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19e0004810 con 0x7f19f00f6510 2023-11-15T06:39:21.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.434+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19f00fc050 con 0x7f19f00f6510 2023-11-15T06:39:21.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.434+0000 7f19e6ffd700 1 -- 172.21.15.135:0/3662506110 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f19e0012380 con 0x7f19f00f6510 2023-11-15T06:39:21.437 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:21.438 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:21.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19dc02ff60 msgr2=0x7f19dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:21.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19dc02ff60 0x7f19dc032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f19d80096a0 tx=0x7f19d8006b40).stop 2023-11-15T06:39:21.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 msgr2=0x7f19f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:21.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f0106730 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f19e0004b70 tx=0x7f19e0004d10).stop 2023-11-15T06:39:21.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 shutdown_connections 2023-11-15T06:39:21.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19dc02ff60 0x7f19dc032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:21.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 --2- 172.21.15.135:0/3662506110 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f19f00f6510 0x7f19f0106730 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:21.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 >> 172.21.15.135:0/3662506110 conn(0x7f19f00f1c70 msgr2=0x7f19f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:21.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 shutdown_connections 2023-11-15T06:39:21.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:21.436+0000 7f19f4bb4700 1 -- 172.21.15.135:0/3662506110 wait complete. 2023-11-15T06:39:21.441 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:22.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:20.245042+0000 mgr.smithi123.razjtn (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:22.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:21.435472+0000 mon.smithi123 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.135:0/3662506110' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:22.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:22.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:23.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.904+0000 7fcf11635700 1 -- 172.21.15.135:0/4249809439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 msgr2=0x7fcf0c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:23.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.904+0000 7fcf11635700 1 --2- 172.21.15.135:0/4249809439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c0f6930 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fcefc009fb0 tx=0x7fcefc009b30).stop 2023-11-15T06:39:23.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.904+0000 7fcf11635700 1 -- 172.21.15.135:0/4249809439 shutdown_connections 2023-11-15T06:39:23.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.904+0000 7fcf11635700 1 --2- 172.21.15.135:0/4249809439 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c0f6930 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:23.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.904+0000 7fcf11635700 1 -- 172.21.15.135:0/4249809439 >> 172.21.15.135:0/4249809439 conn(0x7fcf0c0f1c70 msgr2=0x7fcf0c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:23.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.904+0000 7fcf11635700 1 -- 172.21.15.135:0/4249809439 shutdown_connections 2023-11-15T06:39:23.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.904+0000 7fcf11635700 1 -- 172.21.15.135:0/4249809439 wait complete. 2023-11-15T06:39:23.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.905+0000 7fcf11635700 1 Processor -- start 2023-11-15T06:39:23.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.905+0000 7fcf11635700 1 -- start start 2023-11-15T06:39:23.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.905+0000 7fcf11635700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:23.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.905+0000 7fcf11635700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf0c106bc0 con 0x7fcf0c0f6510 2023-11-15T06:39:23.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcf0affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:23.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcf0affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55898/0 (socket says 172.21.15.135:55898) 2023-11-15T06:39:23.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcf0affd700 1 -- 172.21.15.135:0/2483076162 learned_addr learned my addr 172.21.15.135:0/2483076162 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:23.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcf0affd700 1 -- 172.21.15.135:0/2483076162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcefc005040 con 0x7fcf0c0f6510 2023-11-15T06:39:23.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcf0affd700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c1066d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcefc013510 tx=0x7fcefc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:23.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcef37fe700 1 -- 172.21.15.135:0/2483076162 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcefc009320 con 0x7fcf0c0f6510 2023-11-15T06:39:23.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcef37fe700 1 -- 172.21.15.135:0/2483076162 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcefc013f00 con 0x7fcf0c0f6510 2023-11-15T06:39:23.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf0c106dc0 con 0x7fcf0c0f6510 2023-11-15T06:39:23.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcef37fe700 1 -- 172.21.15.135:0/2483076162 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcefc014d90 con 0x7fcf0c0f6510 2023-11-15T06:39:23.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.906+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf0c107250 con 0x7fcf0c0f6510 2023-11-15T06:39:23.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.907+0000 7fcef37fe700 1 -- 172.21.15.135:0/2483076162 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcefc0263a0 con 0x7fcf0c0f6510 2023-11-15T06:39:23.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.907+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf0c045ad0 con 0x7fcf0c0f6510 2023-11-15T06:39:23.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.907+0000 7fcef37fe700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcef402ff60 0x7fcef4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:23.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.907+0000 7fcef37fe700 1 -- 172.21.15.135:0/2483076162 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcefc0410c0 con 0x7fcf0c0f6510 2023-11-15T06:39:23.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.907+0000 7fcf0a7fc700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcef402ff60 0x7fcef4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:23.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.908+0000 7fcf0a7fc700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcef402ff60 0x7fcef4032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fcf000096a0 tx=0x7fcf00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:23.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:23.912+0000 7fcef37fe700 1 -- 172.21.15.135:0/2483076162 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcefc0095f0 con 0x7fcf0c0f6510 2023-11-15T06:39:24.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.117+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf0c108100 con 0x7fcf0c0f6510 2023-11-15T06:39:24.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.117+0000 7fcef37fe700 1 -- 172.21.15.135:0/2483076162 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcefc01db70 con 0x7fcf0c0f6510 2023-11-15T06:39:24.120 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:24.120 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:24.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcef402ff60 msgr2=0x7fcef4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:24.122 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcef402ff60 0x7fcef4032420 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fcf000096a0 tx=0x7fcf00006b40).stop 2023-11-15T06:39:24.122 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 msgr2=0x7fcf0c1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:24.122 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c1066d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fcefc013510 tx=0x7fcefc00b040).stop 2023-11-15T06:39:24.122 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 shutdown_connections 2023-11-15T06:39:24.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcef402ff60 0x7fcef4032420 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:24.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 --2- 172.21.15.135:0/2483076162 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcf0c0f6510 0x7fcf0c1066d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:24.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 >> 172.21.15.135:0/2483076162 conn(0x7fcf0c0f1c70 msgr2=0x7fcf0c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:24.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 shutdown_connections 2023-11-15T06:39:24.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:24.119+0000 7fcf11635700 1 -- 172.21.15.135:0/2483076162 wait complete. 2023-11-15T06:39:24.124 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:24.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:22.245305+0000 mgr.smithi123.razjtn (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:25.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:24 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:24.118466+0000 mon.smithi123 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.135:0/2483076162' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:25.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:25.503 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:26.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:24.245559+0000 mgr.smithi123.razjtn (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:26.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.683+0000 7f8df4c4d700 1 -- 172.21.15.135:0/246333704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 msgr2=0x7f8df00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:26.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.683+0000 7f8df4c4d700 1 --2- 172.21.15.135:0/246333704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df00f5830 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f8de4009fb0 tx=0x7f8de4009b30).stop 2023-11-15T06:39:26.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.683+0000 7f8df4c4d700 1 -- 172.21.15.135:0/246333704 shutdown_connections 2023-11-15T06:39:26.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.683+0000 7f8df4c4d700 1 --2- 172.21.15.135:0/246333704 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df00f5830 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:26.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.683+0000 7f8df4c4d700 1 -- 172.21.15.135:0/246333704 >> 172.21.15.135:0/246333704 conn(0x7f8df00f0b70 msgr2=0x7f8df00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:26.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.683+0000 7f8df4c4d700 1 -- 172.21.15.135:0/246333704 shutdown_connections 2023-11-15T06:39:26.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.683+0000 7f8df4c4d700 1 -- 172.21.15.135:0/246333704 wait complete. 2023-11-15T06:39:26.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.684+0000 7f8df4c4d700 1 Processor -- start 2023-11-15T06:39:26.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.684+0000 7f8df4c4d700 1 -- start start 2023-11-15T06:39:26.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.684+0000 7f8df4c4d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:26.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.684+0000 7f8df4c4d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8df0105b20 con 0x7f8df00f5410 2023-11-15T06:39:26.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.684+0000 7f8dee59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:26.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.684+0000 7f8dee59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55918/0 (socket says 172.21.15.135:55918) 2023-11-15T06:39:26.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.684+0000 7f8dee59c700 1 -- 172.21.15.135:0/573819016 learned_addr learned my addr 172.21.15.135:0/573819016 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:26.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.685+0000 7f8dee59c700 1 -- 172.21.15.135:0/573819016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8de4005040 con 0x7f8df00f5410 2023-11-15T06:39:26.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.685+0000 7f8dee59c700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df0105630 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8de4013510 tx=0x7f8de400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:26.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.685+0000 7f8ddeffd700 1 -- 172.21.15.135:0/573819016 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de4009320 con 0x7f8df00f5410 2023-11-15T06:39:26.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.685+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8df0105d20 con 0x7f8df00f5410 2023-11-15T06:39:26.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.685+0000 7f8ddeffd700 1 -- 172.21.15.135:0/573819016 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8de4013f00 con 0x7f8df00f5410 2023-11-15T06:39:26.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.685+0000 7f8ddeffd700 1 -- 172.21.15.135:0/573819016 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8de4014d90 con 0x7f8df00f5410 2023-11-15T06:39:26.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.685+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8df01061b0 con 0x7f8df00f5410 2023-11-15T06:39:26.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.686+0000 7f8ddeffd700 1 -- 172.21.15.135:0/573819016 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8de40095f0 con 0x7f8df00f5410 2023-11-15T06:39:26.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.686+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8df0045ad0 con 0x7f8df00f5410 2023-11-15T06:39:26.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.686+0000 7f8ddeffd700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8dd802ff60 0x7f8dd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:26.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.686+0000 7f8ddeffd700 1 -- 172.21.15.135:0/573819016 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8de4042db0 con 0x7f8df00f5410 2023-11-15T06:39:26.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.686+0000 7f8dedd9b700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8dd802ff60 0x7f8dd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:26.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.686+0000 7f8dedd9b700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8dd802ff60 0x7f8dd8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8de00096a0 tx=0x7f8de0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:26.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.690+0000 7f8ddeffd700 1 -- 172.21.15.135:0/573819016 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8de4014420 con 0x7f8df00f5410 2023-11-15T06:39:26.898 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.896+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8df00fae60 con 0x7f8df00f5410 2023-11-15T06:39:26.899 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.897+0000 7f8ddeffd700 1 -- 172.21.15.135:0/573819016 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8de4011290 con 0x7f8df00f5410 2023-11-15T06:39:26.899 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:26.900 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:26.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8dd802ff60 msgr2=0x7f8dd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:26.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8dd802ff60 0x7f8dd8032420 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f8de00096a0 tx=0x7f8de0006b40).stop 2023-11-15T06:39:26.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 msgr2=0x7f8df0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:26.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df0105630 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f8de4013510 tx=0x7f8de400b040).stop 2023-11-15T06:39:26.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 shutdown_connections 2023-11-15T06:39:26.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8dd802ff60 0x7f8dd8032420 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:26.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 --2- 172.21.15.135:0/573819016 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8df00f5410 0x7f8df0105630 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:26.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 >> 172.21.15.135:0/573819016 conn(0x7f8df00f0b70 msgr2=0x7f8df00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:26.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 shutdown_connections 2023-11-15T06:39:26.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:26.898+0000 7f8df4c4d700 1 -- 172.21.15.135:0/573819016 wait complete. 2023-11-15T06:39:26.903 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:28.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:26.245832+0000 mgr.smithi123.razjtn (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:28.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:26.897765+0000 mon.smithi123 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.135:0/573819016' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:28.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:28.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 -- 172.21.15.135:0/3969617745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 msgr2=0x7f55280f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:29.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 --2- 172.21.15.135:0/3969617745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55280f66c0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f5518009fb0 tx=0x7f5518009b30).stop 2023-11-15T06:39:29.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 -- 172.21.15.135:0/3969617745 shutdown_connections 2023-11-15T06:39:29.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 --2- 172.21.15.135:0/3969617745 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55280f66c0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:29.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 -- 172.21.15.135:0/3969617745 >> 172.21.15.135:0/3969617745 conn(0x7f55280f1aa0 msgr2=0x7f55280f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:29.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 -- 172.21.15.135:0/3969617745 shutdown_connections 2023-11-15T06:39:29.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 -- 172.21.15.135:0/3969617745 wait complete. 2023-11-15T06:39:29.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 Processor -- start 2023-11-15T06:39:29.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.433+0000 7f5527fff700 1 -- start start 2023-11-15T06:39:29.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f5527fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55281064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:29.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f5527fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f55281069a0 con 0x7f55280f62a0 2023-11-15T06:39:29.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f5526ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55281064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:29.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f5526ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55281064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53636/0 (socket says 172.21.15.135:53636) 2023-11-15T06:39:29.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f5526ffd700 1 -- 172.21.15.135:0/4010997851 learned_addr learned my addr 172.21.15.135:0/4010997851 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:29.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f5526ffd700 1 -- 172.21.15.135:0/4010997851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5518005040 con 0x7f55280f62a0 2023-11-15T06:39:29.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f5526ffd700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55281064b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5518013510 tx=0x7f551800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:29.445 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.434+0000 7f550f7fe700 1 -- 172.21.15.135:0/4010997851 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5518009320 con 0x7f55280f62a0 2023-11-15T06:39:29.445 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.435+0000 7f550f7fe700 1 -- 172.21.15.135:0/4010997851 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5518013f00 con 0x7f55280f62a0 2023-11-15T06:39:29.445 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.435+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5528106ba0 con 0x7f55280f62a0 2023-11-15T06:39:29.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.435+0000 7f550f7fe700 1 -- 172.21.15.135:0/4010997851 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5518014dc0 con 0x7f55280f62a0 2023-11-15T06:39:29.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.435+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5528107030 con 0x7f55280f62a0 2023-11-15T06:39:29.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.435+0000 7f550f7fe700 1 -- 172.21.15.135:0/4010997851 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f55180095f0 con 0x7f55280f62a0 2023-11-15T06:39:29.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.435+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5528045ad0 con 0x7f55280f62a0 2023-11-15T06:39:29.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.435+0000 7f550f7fe700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f551002ff60 0x7f5510032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:29.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.436+0000 7f550f7fe700 1 -- 172.21.15.135:0/4010997851 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5518041040 con 0x7f55280f62a0 2023-11-15T06:39:29.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.436+0000 7f55267fc700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f551002ff60 0x7f5510032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:29.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.436+0000 7f55267fc700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f551002ff60 0x7f5510032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f551c0096a0 tx=0x7f551c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:29.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.440+0000 7f550f7fe700 1 -- 172.21.15.135:0/4010997851 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5518014420 con 0x7f55280f62a0 2023-11-15T06:39:29.649 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.647+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5528107ee0 con 0x7f55280f62a0 2023-11-15T06:39:29.650 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.648+0000 7f550f7fe700 1 -- 172.21.15.135:0/4010997851 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5518010230 con 0x7f55280f62a0 2023-11-15T06:39:29.651 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:29.651 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:29.652 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f551002ff60 msgr2=0x7f5510032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:29.652 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f551002ff60 0x7f5510032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f551c0096a0 tx=0x7f551c006b40).stop 2023-11-15T06:39:29.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 msgr2=0x7f55281064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:29.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55281064b0 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5518013510 tx=0x7f551800b040).stop 2023-11-15T06:39:29.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 shutdown_connections 2023-11-15T06:39:29.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f551002ff60 0x7f5510032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:29.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 --2- 172.21.15.135:0/4010997851 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f55280f62a0 0x7f55281064b0 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:29.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.649+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 >> 172.21.15.135:0/4010997851 conn(0x7f55280f1aa0 msgr2=0x7f55280f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:29.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.650+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 shutdown_connections 2023-11-15T06:39:29.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:29.650+0000 7f5527fff700 1 -- 172.21.15.135:0/4010997851 wait complete. 2023-11-15T06:39:29.654 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:30.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:28.246130+0000 mgr.smithi123.razjtn (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:30.158 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:29.648689+0000 mon.smithi123 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.135:0/4010997851' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:31.030 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:31.031 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:32.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.136+0000 7f483b61f700 1 -- 172.21.15.135:0/3418359576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 msgr2=0x7f48340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:32.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.136+0000 7f483b61f700 1 --2- 172.21.15.135:0/3418359576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f48340f6930 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f4828009fb0 tx=0x7f4828009b30).stop 2023-11-15T06:39:32.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.136+0000 7f483b61f700 1 -- 172.21.15.135:0/3418359576 shutdown_connections 2023-11-15T06:39:32.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.136+0000 7f483b61f700 1 --2- 172.21.15.135:0/3418359576 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f48340f6930 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:32.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.136+0000 7f483b61f700 1 -- 172.21.15.135:0/3418359576 >> 172.21.15.135:0/3418359576 conn(0x7f48340f1c70 msgr2=0x7f48340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:32.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.136+0000 7f483b61f700 1 -- 172.21.15.135:0/3418359576 shutdown_connections 2023-11-15T06:39:32.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.136+0000 7f483b61f700 1 -- 172.21.15.135:0/3418359576 wait complete. 2023-11-15T06:39:32.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.137+0000 7f483b61f700 1 Processor -- start 2023-11-15T06:39:32.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.137+0000 7f483b61f700 1 -- start start 2023-11-15T06:39:32.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.137+0000 7f483b61f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f4834106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:32.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.137+0000 7f483b61f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4834106c20 con 0x7f48340f6510 2023-11-15T06:39:32.143 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.137+0000 7f48393bb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f4834106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.137+0000 7f48393bb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f4834106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53652/0 (socket says 172.21.15.135:53652) 2023-11-15T06:39:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.137+0000 7f48393bb700 1 -- 172.21.15.135:0/3296690642 learned_addr learned my addr 172.21.15.135:0/3296690642 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.138+0000 7f48393bb700 1 -- 172.21.15.135:0/3296690642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4828005040 con 0x7f48340f6510 2023-11-15T06:39:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.138+0000 7f48393bb700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f4834106730 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f4828013510 tx=0x7f482800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:32.144 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.138+0000 7f4825ffb700 1 -- 172.21.15.135:0/3296690642 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4828009320 con 0x7f48340f6510 2023-11-15T06:39:32.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.138+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4834106e20 con 0x7f48340f6510 2023-11-15T06:39:32.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.138+0000 7f4825ffb700 1 -- 172.21.15.135:0/3296690642 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4828013f00 con 0x7f48340f6510 2023-11-15T06:39:32.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.138+0000 7f4825ffb700 1 -- 172.21.15.135:0/3296690642 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4828014d90 con 0x7f48340f6510 2023-11-15T06:39:32.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.138+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48341072b0 con 0x7f48340f6510 2023-11-15T06:39:32.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.139+0000 7f4825ffb700 1 -- 172.21.15.135:0/3296690642 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f48280095f0 con 0x7f48340f6510 2023-11-15T06:39:32.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.139+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4834045ad0 con 0x7f48340f6510 2023-11-15T06:39:32.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.139+0000 7f4825ffb700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f482002ff60 0x7f4820032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:32.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.139+0000 7f4825ffb700 1 -- 172.21.15.135:0/3296690642 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f48280410c0 con 0x7f48340f6510 2023-11-15T06:39:32.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.139+0000 7f4838bba700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f482002ff60 0x7f4820032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:32.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.140+0000 7f4838bba700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f482002ff60 0x7f4820032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f48300096a0 tx=0x7f4830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:32.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.143+0000 7f4825ffb700 1 -- 172.21.15.135:0/3296690642 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4828014420 con 0x7f48340f6510 2023-11-15T06:39:32.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:30.246507+0000 mgr.smithi123.razjtn (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:32.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.356+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48340fbf60 con 0x7f48340f6510 2023-11-15T06:39:32.359 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.356+0000 7f4825ffb700 1 -- 172.21.15.135:0/3296690642 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f482801d7f0 con 0x7f48340f6510 2023-11-15T06:39:32.359 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:32.360 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:32.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f482002ff60 msgr2=0x7f4820032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:32.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f482002ff60 0x7f4820032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f48300096a0 tx=0x7f4830006b40).stop 2023-11-15T06:39:32.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 msgr2=0x7f4834106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:32.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f4834106730 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f4828013510 tx=0x7f482800b040).stop 2023-11-15T06:39:32.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 shutdown_connections 2023-11-15T06:39:32.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f482002ff60 0x7f4820032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:32.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 --2- 172.21.15.135:0/3296690642 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f48340f6510 0x7f4834106730 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:32.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 >> 172.21.15.135:0/3296690642 conn(0x7f48340f1c70 msgr2=0x7f48340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:32.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 shutdown_connections 2023-11-15T06:39:32.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:32.358+0000 7f483b61f700 1 -- 172.21.15.135:0/3296690642 wait complete. 2023-11-15T06:39:32.363 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:33.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:32.357388+0000 mon.smithi123 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.135:0/3296690642' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:33.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:33.772 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:34.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:32.246917+0000 mgr.smithi123.razjtn (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:34.885 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.881+0000 7f2b0e26f700 1 -- 172.21.15.135:0/460501906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 msgr2=0x7f2b080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:34.886 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.881+0000 7f2b0e26f700 1 --2- 172.21.15.135:0/460501906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b080f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7f2af8009fb0 tx=0x7f2af8009b30).stop 2023-11-15T06:39:34.886 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.881+0000 7f2b0e26f700 1 -- 172.21.15.135:0/460501906 shutdown_connections 2023-11-15T06:39:34.886 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.881+0000 7f2b0e26f700 1 --2- 172.21.15.135:0/460501906 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b080f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:34.886 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.881+0000 7f2b0e26f700 1 -- 172.21.15.135:0/460501906 >> 172.21.15.135:0/460501906 conn(0x7f2b080f0b70 msgr2=0x7f2b080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:34.886 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.881+0000 7f2b0e26f700 1 -- 172.21.15.135:0/460501906 shutdown_connections 2023-11-15T06:39:34.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.881+0000 7f2b0e26f700 1 -- 172.21.15.135:0/460501906 wait complete. 2023-11-15T06:39:34.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.882+0000 7f2b0e26f700 1 Processor -- start 2023-11-15T06:39:34.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.882+0000 7f2b0e26f700 1 -- start start 2023-11-15T06:39:34.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.882+0000 7f2b0e26f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b08109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:34.888 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.882+0000 7f2b0e26f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b08060240 con 0x7f2b080f5410 2023-11-15T06:39:34.889 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.882+0000 7f2b07fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b08109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:34.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.882+0000 7f2b07fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b08109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53668/0 (socket says 172.21.15.135:53668) 2023-11-15T06:39:34.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.882+0000 7f2b07fff700 1 -- 172.21.15.135:0/2913457465 learned_addr learned my addr 172.21.15.135:0/2913457465 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:34.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.883+0000 7f2b07fff700 1 -- 172.21.15.135:0/2913457465 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2af8005040 con 0x7f2b080f5410 2023-11-15T06:39:34.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.883+0000 7f2b07fff700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b08109430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2af8013510 tx=0x7f2af800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:34.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.883+0000 7f2b04ff9700 1 -- 172.21.15.135:0/2913457465 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2af8009320 con 0x7f2b080f5410 2023-11-15T06:39:34.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.883+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b08060440 con 0x7f2b080f5410 2023-11-15T06:39:34.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.883+0000 7f2b04ff9700 1 -- 172.21.15.135:0/2913457465 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2af8013f00 con 0x7f2b080f5410 2023-11-15T06:39:34.892 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.883+0000 7f2b04ff9700 1 -- 172.21.15.135:0/2913457465 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2af8014d90 con 0x7f2b080f5410 2023-11-15T06:39:34.892 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.883+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b08109ae0 con 0x7f2b080f5410 2023-11-15T06:39:34.892 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.884+0000 7f2b04ff9700 1 -- 172.21.15.135:0/2913457465 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2af80095f0 con 0x7f2b080f5410 2023-11-15T06:39:34.892 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.884+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b08045ad0 con 0x7f2b080f5410 2023-11-15T06:39:34.892 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.884+0000 7f2b04ff9700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2af002ff60 0x7f2af0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:34.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.884+0000 7f2b04ff9700 1 -- 172.21.15.135:0/2913457465 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2af80410c0 con 0x7f2b080f5410 2023-11-15T06:39:34.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.884+0000 7f2b077fe700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2af002ff60 0x7f2af0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:34.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.884+0000 7f2b077fe700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2af002ff60 0x7f2af0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2afc0096a0 tx=0x7f2afc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:34.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:34.888+0000 7f2b04ff9700 1 -- 172.21.15.135:0/2913457465 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2af8014420 con 0x7f2b080f5410 2023-11-15T06:39:35.095 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.092+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b080fae60 con 0x7f2b080f5410 2023-11-15T06:39:35.096 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.092+0000 7f2b04ff9700 1 -- 172.21.15.135:0/2913457465 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2af8010230 con 0x7f2b080f5410 2023-11-15T06:39:35.097 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:35.097 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:35.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2af002ff60 msgr2=0x7f2af0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:35.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2af002ff60 0x7f2af0032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f2afc0096a0 tx=0x7f2afc006b40).stop 2023-11-15T06:39:35.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 msgr2=0x7f2b08109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:35.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b08109430 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f2af8013510 tx=0x7f2af800b040).stop 2023-11-15T06:39:35.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 shutdown_connections 2023-11-15T06:39:35.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2af002ff60 0x7f2af0032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:35.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 --2- 172.21.15.135:0/2913457465 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2b080f5410 0x7f2b08109430 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:35.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 >> 172.21.15.135:0/2913457465 conn(0x7f2b080f0b70 msgr2=0x7f2b080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:35.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 shutdown_connections 2023-11-15T06:39:35.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:35.094+0000 7f2b0e26f700 1 -- 172.21.15.135:0/2913457465 wait complete. 2023-11-15T06:39:35.101 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:36.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:34.247287+0000 mgr.smithi123.razjtn (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:36.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:35.094504+0000 mon.smithi123 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.135:0/2913457465' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:36.480 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:36.481 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:37.606 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.598+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3838356087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 msgr2=0x7f5b440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:37.606 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.598+0000 7f5b4b9a1700 1 --2- 172.21.15.135:0/3838356087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b440f6930 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f5b3c009960 tx=0x7f5b3c004d10).stop 2023-11-15T06:39:37.606 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.598+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3838356087 shutdown_connections 2023-11-15T06:39:37.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.598+0000 7f5b4b9a1700 1 --2- 172.21.15.135:0/3838356087 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b440f6930 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:37.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.598+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3838356087 >> 172.21.15.135:0/3838356087 conn(0x7f5b440f1c70 msgr2=0x7f5b440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:37.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.598+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3838356087 shutdown_connections 2023-11-15T06:39:37.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.598+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3838356087 wait complete. 2023-11-15T06:39:37.607 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.599+0000 7f5b4b9a1700 1 Processor -- start 2023-11-15T06:39:37.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.599+0000 7f5b4b9a1700 1 -- start start 2023-11-15T06:39:37.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.599+0000 7f5b4b9a1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b44106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:37.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.599+0000 7f5b4b9a1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b44106c20 con 0x7f5b440f6510 2023-11-15T06:39:37.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.599+0000 7f5b4973d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b44106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:37.608 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.599+0000 7f5b4973d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b44106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53672/0 (socket says 172.21.15.135:53672) 2023-11-15T06:39:37.609 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.599+0000 7f5b4973d700 1 -- 172.21.15.135:0/3757319272 learned_addr learned my addr 172.21.15.135:0/3757319272 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:37.609 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.600+0000 7f5b4973d700 1 -- 172.21.15.135:0/3757319272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b3c009640 con 0x7f5b440f6510 2023-11-15T06:39:37.609 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.600+0000 7f5b4973d700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b44106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f5b3c004b70 tx=0x7f5b3c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:37.609 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.600+0000 7f5b41ffb700 1 -- 172.21.15.135:0/3757319272 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b3c018070 con 0x7f5b440f6510 2023-11-15T06:39:37.609 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.600+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b44106e20 con 0x7f5b440f6510 2023-11-15T06:39:37.610 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.600+0000 7f5b41ffb700 1 -- 172.21.15.135:0/3757319272 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b3c00b870 con 0x7f5b440f6510 2023-11-15T06:39:37.610 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.600+0000 7f5b41ffb700 1 -- 172.21.15.135:0/3757319272 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b3c01c560 con 0x7f5b440f6510 2023-11-15T06:39:37.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.600+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b441072b0 con 0x7f5b440f6510 2023-11-15T06:39:37.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.601+0000 7f5b41ffb700 1 -- 172.21.15.135:0/3757319272 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b3c013630 con 0x7f5b440f6510 2023-11-15T06:39:37.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.601+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b44045ad0 con 0x7f5b440f6510 2023-11-15T06:39:37.611 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.601+0000 7f5b41ffb700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b2c02ff60 0x7f5b2c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:37.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.601+0000 7f5b41ffb700 1 -- 172.21.15.135:0/3757319272 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b3c041290 con 0x7f5b440f6510 2023-11-15T06:39:37.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.601+0000 7f5b48f3c700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b2c02ff60 0x7f5b2c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:37.612 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.602+0000 7f5b48f3c700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b2c02ff60 0x7f5b2c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5b380096a0 tx=0x7f5b38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:37.613 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.606+0000 7f5b41ffb700 1 -- 172.21.15.135:0/3757319272 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b3c004810 con 0x7f5b440f6510 2023-11-15T06:39:37.818 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.815+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b440fbf60 con 0x7f5b440f6510 2023-11-15T06:39:37.819 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.815+0000 7f5b41ffb700 1 -- 172.21.15.135:0/3757319272 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b3c012380 con 0x7f5b440f6510 2023-11-15T06:39:37.820 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:37.820 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:37.822 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b2c02ff60 msgr2=0x7f5b2c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:37.823 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b2c02ff60 0x7f5b2c032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f5b380096a0 tx=0x7f5b38006b40).stop 2023-11-15T06:39:37.823 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 msgr2=0x7f5b44106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:37.823 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b44106730 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f5b3c004b70 tx=0x7f5b3c004d10).stop 2023-11-15T06:39:37.823 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 shutdown_connections 2023-11-15T06:39:37.824 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b2c02ff60 0x7f5b2c032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:37.824 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 --2- 172.21.15.135:0/3757319272 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b440f6510 0x7f5b44106730 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:37.824 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 >> 172.21.15.135:0/3757319272 conn(0x7f5b440f1c70 msgr2=0x7f5b440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:37.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 shutdown_connections 2023-11-15T06:39:37.825 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:37.817+0000 7f5b4b9a1700 1 -- 172.21.15.135:0/3757319272 wait complete. 2023-11-15T06:39:37.825 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:38.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:36.247538+0000 mgr.smithi123.razjtn (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:38.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:37.817406+0000 mon.smithi123 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.135:0/3757319272' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:39.204 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:39.205 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:40.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:38.247818+0000 mgr.smithi123.razjtn (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:40.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.298+0000 7f40cfad7700 1 -- 172.21.15.135:0/2555277848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 msgr2=0x7f40c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:40.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.298+0000 7f40cfad7700 1 --2- 172.21.15.135:0/2555277848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c80f6930 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f40b8009fb0 tx=0x7f40b8009b30).stop 2023-11-15T06:39:40.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.298+0000 7f40cfad7700 1 -- 172.21.15.135:0/2555277848 shutdown_connections 2023-11-15T06:39:40.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.298+0000 7f40cfad7700 1 --2- 172.21.15.135:0/2555277848 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c80f6930 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:40.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.298+0000 7f40cfad7700 1 -- 172.21.15.135:0/2555277848 >> 172.21.15.135:0/2555277848 conn(0x7f40c80f1c70 msgr2=0x7f40c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:40.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.298+0000 7f40cfad7700 1 -- 172.21.15.135:0/2555277848 shutdown_connections 2023-11-15T06:39:40.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.298+0000 7f40cfad7700 1 -- 172.21.15.135:0/2555277848 wait complete. 2023-11-15T06:39:40.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.299+0000 7f40cfad7700 1 Processor -- start 2023-11-15T06:39:40.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.299+0000 7f40cfad7700 1 -- start start 2023-11-15T06:39:40.306 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.299+0000 7f40cfad7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:40.306 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.299+0000 7f40cfad7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40c8106c20 con 0x7f40c80f6510 2023-11-15T06:39:40.306 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.299+0000 7f40cd873700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:40.306 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.299+0000 7f40cd873700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48308/0 (socket says 172.21.15.135:48308) 2023-11-15T06:39:40.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.299+0000 7f40cd873700 1 -- 172.21.15.135:0/83823266 learned_addr learned my addr 172.21.15.135:0/83823266 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:40.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.300+0000 7f40cd873700 1 -- 172.21.15.135:0/83823266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b8005040 con 0x7f40c80f6510 2023-11-15T06:39:40.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.300+0000 7f40cd873700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c8106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f40b8013510 tx=0x7f40b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:40.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.300+0000 7f40be7fc700 1 -- 172.21.15.135:0/83823266 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b8009320 con 0x7f40c80f6510 2023-11-15T06:39:40.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.300+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40c8106e20 con 0x7f40c80f6510 2023-11-15T06:39:40.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.300+0000 7f40be7fc700 1 -- 172.21.15.135:0/83823266 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f40b8013f00 con 0x7f40c80f6510 2023-11-15T06:39:40.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.300+0000 7f40be7fc700 1 -- 172.21.15.135:0/83823266 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40b8014d90 con 0x7f40c80f6510 2023-11-15T06:39:40.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.300+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40c81072b0 con 0x7f40c80f6510 2023-11-15T06:39:40.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.301+0000 7f40be7fc700 1 -- 172.21.15.135:0/83823266 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f40b80095f0 con 0x7f40c80f6510 2023-11-15T06:39:40.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.301+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40c8045ad0 con 0x7f40c80f6510 2023-11-15T06:39:40.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.301+0000 7f40be7fc700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f40b402ff60 0x7f40b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:40.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.301+0000 7f40be7fc700 1 -- 172.21.15.135:0/83823266 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40b8040fd0 con 0x7f40c80f6510 2023-11-15T06:39:40.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.301+0000 7f40cd072700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f40b402ff60 0x7f40b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:40.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.302+0000 7f40cd072700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f40b402ff60 0x7f40b4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f40c40096a0 tx=0x7f40c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:40.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.306+0000 7f40be7fc700 1 -- 172.21.15.135:0/83823266 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40b8014420 con 0x7f40c80f6510 2023-11-15T06:39:40.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.515+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f40c80fbf60 con 0x7f40c80f6510 2023-11-15T06:39:40.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.515+0000 7f40be7fc700 1 -- 172.21.15.135:0/83823266 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f40b801d7f0 con 0x7f40c80f6510 2023-11-15T06:39:40.519 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:40.520 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:40.521 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f40b402ff60 msgr2=0x7f40b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:40.521 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f40b402ff60 0x7f40b4032420 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f40c40096a0 tx=0x7f40c4006b40).stop 2023-11-15T06:39:40.521 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 msgr2=0x7f40c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:40.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c8106730 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f40b8013510 tx=0x7f40b800b040).stop 2023-11-15T06:39:40.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 shutdown_connections 2023-11-15T06:39:40.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f40b402ff60 0x7f40b4032420 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:40.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 --2- 172.21.15.135:0/83823266 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40c80f6510 0x7f40c8106730 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:40.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 >> 172.21.15.135:0/83823266 conn(0x7f40c80f1c70 msgr2=0x7f40c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:40.523 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 shutdown_connections 2023-11-15T06:39:40.523 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:40.517+0000 7f40cfad7700 1 -- 172.21.15.135:0/83823266 wait complete. 2023-11-15T06:39:40.523 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:41.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:40.363455+0000 mon.smithi123 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:39:41.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:40.370486+0000 mon.smithi123 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:39:41.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:40.517538+0000 mon.smithi123 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.135:0/83823266' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:41.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:41.891 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:42.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:40.248098+0000 mgr.smithi123.razjtn (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:42.987 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 -- 172.21.15.135:0/4148182161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 msgr2=0x7f82640f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:42.987 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 --2- 172.21.15.135:0/4148182161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f82640f55c0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f8254009fb0 tx=0x7f8254009b30).stop 2023-11-15T06:39:42.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 -- 172.21.15.135:0/4148182161 shutdown_connections 2023-11-15T06:39:42.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 --2- 172.21.15.135:0/4148182161 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f82640f55c0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:42.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 -- 172.21.15.135:0/4148182161 >> 172.21.15.135:0/4148182161 conn(0x7f82640f09a0 msgr2=0x7f82640f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:42.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 -- 172.21.15.135:0/4148182161 shutdown_connections 2023-11-15T06:39:42.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 -- 172.21.15.135:0/4148182161 wait complete. 2023-11-15T06:39:42.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 Processor -- start 2023-11-15T06:39:42.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 -- start start 2023-11-15T06:39:42.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.983+0000 7f8269746700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f826410b5d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:42.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8269746700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8264106fc0 con 0x7f82640f51a0 2023-11-15T06:39:42.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8263fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f826410b5d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:42.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8263fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f826410b5d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48336/0 (socket says 172.21.15.135:48336) 2023-11-15T06:39:42.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8263fff700 1 -- 172.21.15.135:0/41399695 learned_addr learned my addr 172.21.15.135:0/41399695 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:42.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8263fff700 1 -- 172.21.15.135:0/41399695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8254005040 con 0x7f82640f51a0 2023-11-15T06:39:42.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8263fff700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f826410b5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8254013510 tx=0x7f825400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:42.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8260ff9700 1 -- 172.21.15.135:0/41399695 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8254009320 con 0x7f82640f51a0 2023-11-15T06:39:42.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8264060220 con 0x7f82640f51a0 2023-11-15T06:39:42.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8260ff9700 1 -- 172.21.15.135:0/41399695 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8254013f00 con 0x7f82640f51a0 2023-11-15T06:39:42.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8260ff9700 1 -- 172.21.15.135:0/41399695 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8254014dc0 con 0x7f82640f51a0 2023-11-15T06:39:42.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.984+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f826410bac0 con 0x7f82640f51a0 2023-11-15T06:39:42.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.985+0000 7f8260ff9700 1 -- 172.21.15.135:0/41399695 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f825401d3d0 con 0x7f82640f51a0 2023-11-15T06:39:42.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.985+0000 7f8260ff9700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f824c02ff60 0x7f824c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:42.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.985+0000 7f8260ff9700 1 -- 172.21.15.135:0/41399695 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f82540410e0 con 0x7f82640f51a0 2023-11-15T06:39:42.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.985+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8264059100 con 0x7f82640f51a0 2023-11-15T06:39:42.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.985+0000 7f82637fe700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f824c02ff60 0x7f824c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:42.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.986+0000 7f82637fe700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f824c02ff60 0x7f824c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f82580096a0 tx=0x7f8258006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:42.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:42.991+0000 7f8260ff9700 1 -- 172.21.15.135:0/41399695 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8254012920 con 0x7f82640f51a0 2023-11-15T06:39:43.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.200+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8264045ad0 con 0x7f82640f51a0 2023-11-15T06:39:43.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.200+0000 7f8260ff9700 1 -- 172.21.15.135:0/41399695 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f825402b020 con 0x7f82640f51a0 2023-11-15T06:39:43.205 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:43.205 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:43.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f824c02ff60 msgr2=0x7f824c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:43.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f824c02ff60 0x7f824c032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f82580096a0 tx=0x7f8258006b40).stop 2023-11-15T06:39:43.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 msgr2=0x7f826410b5d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:43.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f826410b5d0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8254013510 tx=0x7f825400b040).stop 2023-11-15T06:39:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 shutdown_connections 2023-11-15T06:39:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f824c02ff60 0x7f824c032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 --2- 172.21.15.135:0/41399695 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f82640f51a0 0x7f826410b5d0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 >> 172.21.15.135:0/41399695 conn(0x7f82640f09a0 msgr2=0x7f82640f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 shutdown_connections 2023-11-15T06:39:43.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:43.202+0000 7f8269746700 1 -- 172.21.15.135:0/41399695 wait complete. 2023-11-15T06:39:43.208 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:44.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:42.248368+0000 mgr.smithi123.razjtn (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:44.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:43.202378+0000 mon.smithi123 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.135:0/41399695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:44.585 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:44.586 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:45.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 -- 172.21.15.135:0/3146376739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 msgr2=0x7f164c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:45.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 --2- 172.21.15.135:0/3146376739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c0f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f163c009fb0 tx=0x7f163c009b30).stop 2023-11-15T06:39:45.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 -- 172.21.15.135:0/3146376739 shutdown_connections 2023-11-15T06:39:45.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 --2- 172.21.15.135:0/3146376739 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c0f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:45.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 -- 172.21.15.135:0/3146376739 >> 172.21.15.135:0/3146376739 conn(0x7f164c0f0b70 msgr2=0x7f164c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:45.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 -- 172.21.15.135:0/3146376739 shutdown_connections 2023-11-15T06:39:45.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 -- 172.21.15.135:0/3146376739 wait complete. 2023-11-15T06:39:45.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.725+0000 7f16546fe700 1 Processor -- start 2023-11-15T06:39:45.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f16546fe700 1 -- start start 2023-11-15T06:39:45.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f16546fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:45.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f16546fe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f164c060240 con 0x7f164c0f5410 2023-11-15T06:39:45.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f165249a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:45.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f165249a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48350/0 (socket says 172.21.15.135:48350) 2023-11-15T06:39:45.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f165249a700 1 -- 172.21.15.135:0/1066522160 learned_addr learned my addr 172.21.15.135:0/1066522160 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:45.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f165249a700 1 -- 172.21.15.135:0/1066522160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f163c005040 con 0x7f164c0f5410 2023-11-15T06:39:45.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.726+0000 7f165249a700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c109430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f163c013510 tx=0x7f163c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:45.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.727+0000 7f1642ffd700 1 -- 172.21.15.135:0/1066522160 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f163c009320 con 0x7f164c0f5410 2023-11-15T06:39:45.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.727+0000 7f1642ffd700 1 -- 172.21.15.135:0/1066522160 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f163c013f00 con 0x7f164c0f5410 2023-11-15T06:39:45.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.727+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f164c060440 con 0x7f164c0f5410 2023-11-15T06:39:45.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.727+0000 7f1642ffd700 1 -- 172.21.15.135:0/1066522160 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f163c014d90 con 0x7f164c0f5410 2023-11-15T06:39:45.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.727+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f164c109ae0 con 0x7f164c0f5410 2023-11-15T06:39:45.739 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.728+0000 7f1642ffd700 1 -- 172.21.15.135:0/1066522160 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f163c0095f0 con 0x7f164c0f5410 2023-11-15T06:39:45.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.728+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f164c045ad0 con 0x7f164c0f5410 2023-11-15T06:39:45.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.728+0000 7f1642ffd700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f163802ff60 0x7f1638032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:45.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.728+0000 7f1642ffd700 1 -- 172.21.15.135:0/1066522160 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f163c041000 con 0x7f164c0f5410 2023-11-15T06:39:45.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.728+0000 7f1651c99700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f163802ff60 0x7f1638032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:45.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.729+0000 7f1651c99700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f163802ff60 0x7f1638032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f16480096a0 tx=0x7f1648006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:45.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.733+0000 7f1642ffd700 1 -- 172.21.15.135:0/1066522160 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f163c014420 con 0x7f164c0f5410 2023-11-15T06:39:45.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.939+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f164c0fae60 con 0x7f164c0f5410 2023-11-15T06:39:45.944 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.940+0000 7f1642ffd700 1 -- 172.21.15.135:0/1066522160 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f163c01d7f0 con 0x7f164c0f5410 2023-11-15T06:39:45.944 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:45.944 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:45.945 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f163802ff60 msgr2=0x7f1638032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:45.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f163802ff60 0x7f1638032420 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f16480096a0 tx=0x7f1648006b40).stop 2023-11-15T06:39:45.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 msgr2=0x7f164c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:45.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c109430 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f163c013510 tx=0x7f163c00b040).stop 2023-11-15T06:39:45.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 shutdown_connections 2023-11-15T06:39:45.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f163802ff60 0x7f1638032420 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:45.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 --2- 172.21.15.135:0/1066522160 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f164c0f5410 0x7f164c109430 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:45.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 >> 172.21.15.135:0/1066522160 conn(0x7f164c0f0b70 msgr2=0x7f164c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:45.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 shutdown_connections 2023-11-15T06:39:45.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:45.943+0000 7f16546fe700 1 -- 172.21.15.135:0/1066522160 wait complete. 2023-11-15T06:39:45.948 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:46.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:44.248731+0000 mgr.smithi123.razjtn (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:47.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:46 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:45.942222+0000 mon.smithi123 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.135:0/1066522160' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:47.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:47.327 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:48.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:46.249145+0000 mgr.smithi123.razjtn (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:48.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.433+0000 7f9eaf45c700 1 -- 172.21.15.135:0/4138753525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 msgr2=0x7f9ea80f68a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:48.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.433+0000 7f9eaf45c700 1 --2- 172.21.15.135:0/4138753525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea80f68a0 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f9e9c009fb0 tx=0x7f9e9c009b30).stop 2023-11-15T06:39:48.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 -- 172.21.15.135:0/4138753525 shutdown_connections 2023-11-15T06:39:48.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 --2- 172.21.15.135:0/4138753525 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea80f68a0 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:48.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 -- 172.21.15.135:0/4138753525 >> 172.21.15.135:0/4138753525 conn(0x7f9ea80f1c00 msgr2=0x7f9ea80f4020 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:48.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 -- 172.21.15.135:0/4138753525 shutdown_connections 2023-11-15T06:39:48.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 -- 172.21.15.135:0/4138753525 wait complete. 2023-11-15T06:39:48.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 Processor -- start 2023-11-15T06:39:48.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 -- start start 2023-11-15T06:39:48.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea81066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:48.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9eaf45c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ea8106bc0 con 0x7f9ea80f6480 2023-11-15T06:39:48.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.434+0000 7f9ead1f8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea81066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:48.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9ead1f8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea81066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54594/0 (socket says 172.21.15.135:54594) 2023-11-15T06:39:48.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9ead1f8700 1 -- 172.21.15.135:0/3026609075 learned_addr learned my addr 172.21.15.135:0/3026609075 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:48.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9ead1f8700 1 -- 172.21.15.135:0/3026609075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9e9c005040 con 0x7f9ea80f6480 2023-11-15T06:39:48.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9ead1f8700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea81066d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9e9c014000 tx=0x7f9e9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:48.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9e99ffb700 1 -- 172.21.15.135:0/3026609075 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9c0095f0 con 0x7f9ea80f6480 2023-11-15T06:39:48.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ea8106dc0 con 0x7f9ea80f6480 2023-11-15T06:39:48.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9e99ffb700 1 -- 172.21.15.135:0/3026609075 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9e9c007e70 con 0x7f9ea80f6480 2023-11-15T06:39:48.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9e99ffb700 1 -- 172.21.15.135:0/3026609075 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9e9c014ea0 con 0x7f9ea80f6480 2023-11-15T06:39:48.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.435+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ea8107250 con 0x7f9ea80f6480 2023-11-15T06:39:48.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.436+0000 7f9e99ffb700 1 -- 172.21.15.135:0/3026609075 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9e9c025070 con 0x7f9ea80f6480 2023-11-15T06:39:48.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.436+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ea8045ad0 con 0x7f9ea80f6480 2023-11-15T06:39:48.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.436+0000 7f9e99ffb700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9e9402ff60 0x7f9e94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:48.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.436+0000 7f9e99ffb700 1 -- 172.21.15.135:0/3026609075 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9e9c040970 con 0x7f9ea80f6480 2023-11-15T06:39:48.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.437+0000 7f9eac9f7700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9e9402ff60 0x7f9e94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:48.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.437+0000 7f9eac9f7700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9e9402ff60 0x7f9e94032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9ea40096a0 tx=0x7f9ea4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:48.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.441+0000 7f9e99ffb700 1 -- 172.21.15.135:0/3026609075 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9e9c0079c0 con 0x7f9ea80f6480 2023-11-15T06:39:48.650 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.648+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ea8108100 con 0x7f9ea80f6480 2023-11-15T06:39:48.651 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.648+0000 7f9e99ffb700 1 -- 172.21.15.135:0/3026609075 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9e9c00f300 con 0x7f9ea80f6480 2023-11-15T06:39:48.651 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:48.651 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:48.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9e9402ff60 msgr2=0x7f9e94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:48.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9e9402ff60 0x7f9e94032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f9ea40096a0 tx=0x7f9ea4006b40).stop 2023-11-15T06:39:48.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 msgr2=0x7f9ea81066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:48.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea81066d0 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f9e9c014000 tx=0x7f9e9c00b040).stop 2023-11-15T06:39:48.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 shutdown_connections 2023-11-15T06:39:48.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9e9402ff60 0x7f9e94032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:48.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 --2- 172.21.15.135:0/3026609075 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ea80f6480 0x7f9ea81066d0 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:48.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 >> 172.21.15.135:0/3026609075 conn(0x7f9ea80f1c00 msgr2=0x7f9ea80f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:48.654 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 shutdown_connections 2023-11-15T06:39:48.655 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:48.650+0000 7f9eaf45c700 1 -- 172.21.15.135:0/3026609075 wait complete. 2023-11-15T06:39:48.655 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:49.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:48 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:48.649384+0000 mon.smithi123 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.135:0/3026609075' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:50.034 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:50.034 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:50.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:48.249448+0000 mgr.smithi123.razjtn (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:51.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.129+0000 7fce04e38700 1 -- 172.21.15.135:0/2799693947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 msgr2=0x7fce000f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:51.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.129+0000 7fce04e38700 1 --2- 172.21.15.135:0/2799693947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce000f6930 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fcdf0009fb0 tx=0x7fcdf0009b30).stop 2023-11-15T06:39:51.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.129+0000 7fce04e38700 1 -- 172.21.15.135:0/2799693947 shutdown_connections 2023-11-15T06:39:51.134 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.129+0000 7fce04e38700 1 --2- 172.21.15.135:0/2799693947 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce000f6930 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:51.134 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.129+0000 7fce04e38700 1 -- 172.21.15.135:0/2799693947 >> 172.21.15.135:0/2799693947 conn(0x7fce000f1c70 msgr2=0x7fce000f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:51.134 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.130+0000 7fce04e38700 1 -- 172.21.15.135:0/2799693947 shutdown_connections 2023-11-15T06:39:51.134 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.130+0000 7fce04e38700 1 -- 172.21.15.135:0/2799693947 wait complete. 2023-11-15T06:39:51.134 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.130+0000 7fce04e38700 1 Processor -- start 2023-11-15T06:39:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.130+0000 7fce04e38700 1 -- start start 2023-11-15T06:39:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.130+0000 7fce04e38700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce00106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:51.135 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.130+0000 7fce04e38700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce00106c20 con 0x7fce000f6510 2023-11-15T06:39:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdfe59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce00106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdfe59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce00106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54612/0 (socket says 172.21.15.135:54612) 2023-11-15T06:39:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdfe59c700 1 -- 172.21.15.135:0/626946626 learned_addr learned my addr 172.21.15.135:0/626946626 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdfe59c700 1 -- 172.21.15.135:0/626946626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf0005040 con 0x7fce000f6510 2023-11-15T06:39:51.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdfe59c700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce00106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fcdf0014000 tx=0x7fcdf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdf6ffd700 1 -- 172.21.15.135:0/626946626 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdf00095f0 con 0x7fce000f6510 2023-11-15T06:39:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce00106e20 con 0x7fce000f6510 2023-11-15T06:39:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdf6ffd700 1 -- 172.21.15.135:0/626946626 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdf0007e70 con 0x7fce000f6510 2023-11-15T06:39:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fcdf6ffd700 1 -- 172.21.15.135:0/626946626 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdf0014ea0 con 0x7fce000f6510 2023-11-15T06:39:51.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.131+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce001072b0 con 0x7fce000f6510 2023-11-15T06:39:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.132+0000 7fcdf6ffd700 1 -- 172.21.15.135:0/626946626 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcdf0025070 con 0x7fce000f6510 2023-11-15T06:39:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.132+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce00045ad0 con 0x7fce000f6510 2023-11-15T06:39:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.132+0000 7fcdf6ffd700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdec02ff60 0x7fcdec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.132+0000 7fcdf6ffd700 1 -- 172.21.15.135:0/626946626 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcdf0040a30 con 0x7fce000f6510 2023-11-15T06:39:51.139 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.132+0000 7fcdfdd9b700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdec02ff60 0x7fcdec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:51.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.133+0000 7fcdfdd9b700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdec02ff60 0x7fcdec032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcde80096a0 tx=0x7fcde8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:51.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.137+0000 7fcdf6ffd700 1 -- 172.21.15.135:0/626946626 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdf00079c0 con 0x7fce000f6510 2023-11-15T06:39:51.345 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.342+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce000fbf60 con 0x7fce000f6510 2023-11-15T06:39:51.346 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.343+0000 7fcdf6ffd700 1 -- 172.21.15.135:0/626946626 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcdf000f300 con 0x7fce000f6510 2023-11-15T06:39:51.346 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:51.347 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:51.348 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdec02ff60 msgr2=0x7fcdec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:51.348 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdec02ff60 0x7fcdec032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fcde80096a0 tx=0x7fcde8006b40).stop 2023-11-15T06:39:51.349 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 msgr2=0x7fce00106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:51.349 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce00106730 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7fcdf0014000 tx=0x7fcdf000b040).stop 2023-11-15T06:39:51.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 shutdown_connections 2023-11-15T06:39:51.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdec02ff60 0x7fcdec032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:51.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 --2- 172.21.15.135:0/626946626 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fce000f6510 0x7fce00106730 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:51.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 >> 172.21.15.135:0/626946626 conn(0x7fce000f1c70 msgr2=0x7fce000f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:51.352 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 shutdown_connections 2023-11-15T06:39:51.352 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:51.344+0000 7fce04e38700 1 -- 172.21.15.135:0/626946626 wait complete. 2023-11-15T06:39:51.352 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:52.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:50.249769+0000 mgr.smithi123.razjtn (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:52.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:51.343653+0000 mon.smithi123 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.135:0/626946626' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:52.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:52.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:53.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.837+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/4156008628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 msgr2=0x7f4a880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:53.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.837+0000 7f4a8e0c7700 1 --2- 172.21.15.135:0/4156008628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a880f5830 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f4a7c009fb0 tx=0x7f4a7c009b30).stop 2023-11-15T06:39:53.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.837+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/4156008628 shutdown_connections 2023-11-15T06:39:53.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.837+0000 7f4a8e0c7700 1 --2- 172.21.15.135:0/4156008628 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a880f5830 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:53.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.837+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/4156008628 >> 172.21.15.135:0/4156008628 conn(0x7f4a880f0b70 msgr2=0x7f4a880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:53.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.837+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/4156008628 shutdown_connections 2023-11-15T06:39:53.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.837+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/4156008628 wait complete. 2023-11-15T06:39:53.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a8e0c7700 1 Processor -- start 2023-11-15T06:39:53.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a8e0c7700 1 -- start start 2023-11-15T06:39:53.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a8e0c7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a88109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:53.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a8e0c7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4a88060240 con 0x7f4a880f5410 2023-11-15T06:39:53.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a877fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a88109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:53.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a877fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a88109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54628/0 (socket says 172.21.15.135:54628) 2023-11-15T06:39:53.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a877fe700 1 -- 172.21.15.135:0/774029166 learned_addr learned my addr 172.21.15.135:0/774029166 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:53.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.838+0000 7f4a877fe700 1 -- 172.21.15.135:0/774029166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4a7c005040 con 0x7f4a880f5410 2023-11-15T06:39:53.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.839+0000 7f4a877fe700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a88109400 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4a7c013510 tx=0x7f4a7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:53.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.839+0000 7f4a6ffff700 1 -- 172.21.15.135:0/774029166 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a7c009320 con 0x7f4a880f5410 2023-11-15T06:39:53.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.839+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4a88060440 con 0x7f4a880f5410 2023-11-15T06:39:53.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.839+0000 7f4a6ffff700 1 -- 172.21.15.135:0/774029166 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4a7c013f00 con 0x7f4a880f5410 2023-11-15T06:39:53.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.839+0000 7f4a6ffff700 1 -- 172.21.15.135:0/774029166 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4a7c014d90 con 0x7f4a880f5410 2023-11-15T06:39:53.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.839+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4a88109ab0 con 0x7f4a880f5410 2023-11-15T06:39:53.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.839+0000 7f4a6ffff700 1 -- 172.21.15.135:0/774029166 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4a7c0263a0 con 0x7f4a880f5410 2023-11-15T06:39:53.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.840+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4a88045ad0 con 0x7f4a880f5410 2023-11-15T06:39:53.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.840+0000 7f4a6ffff700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:53.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.840+0000 7f4a6ffff700 1 -- 172.21.15.135:0/774029166 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4a7c041380 con 0x7f4a880f5410 2023-11-15T06:39:53.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.840+0000 7f4a86ffd700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:53.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.840+0000 7f4a86ffd700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4a7002ff60 0x7f4a70032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4a780096a0 tx=0x7f4a78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:53.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:53.844+0000 7f4a6ffff700 1 -- 172.21.15.135:0/774029166 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4a7c01cac0 con 0x7f4a880f5410 2023-11-15T06:39:54.052 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.049+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4a880fae60 con 0x7f4a880f5410 2023-11-15T06:39:54.053 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.050+0000 7f4a6ffff700 1 -- 172.21.15.135:0/774029166 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4a7c01cac0 con 0x7f4a880f5410 2023-11-15T06:39:54.053 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:54.053 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:54.054 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4a7002ff60 msgr2=0x7f4a70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:54.054 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4a7002ff60 0x7f4a70032420 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f4a780096a0 tx=0x7f4a78006b40).stop 2023-11-15T06:39:54.054 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 msgr2=0x7f4a88109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:54.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a88109400 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f4a7c013510 tx=0x7f4a7c00b040).stop 2023-11-15T06:39:54.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 shutdown_connections 2023-11-15T06:39:54.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4a7002ff60 0x7f4a70032420 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:54.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 --2- 172.21.15.135:0/774029166 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4a880f5410 0x7f4a88109400 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:54.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 >> 172.21.15.135:0/774029166 conn(0x7f4a880f0b70 msgr2=0x7f4a880f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:54.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 shutdown_connections 2023-11-15T06:39:54.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:54.051+0000 7f4a8e0c7700 1 -- 172.21.15.135:0/774029166 wait complete. 2023-11-15T06:39:54.056 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:54.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:52.250109+0000 mgr.smithi123.razjtn (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:55.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:54 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:54.050869+0000 mon.smithi123 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.135:0/774029166' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:55.440 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:55.440 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:56.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:54.250378+0000 mgr.smithi123.razjtn (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:56.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 -- 172.21.15.135:0/2604332436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 msgr2=0x7f46500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:56.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 --2- 172.21.15.135:0/2604332436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46500f55c0 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f4640009fb0 tx=0x7f4640009b30).stop 2023-11-15T06:39:56.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 -- 172.21.15.135:0/2604332436 shutdown_connections 2023-11-15T06:39:56.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 --2- 172.21.15.135:0/2604332436 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46500f55c0 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:56.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 -- 172.21.15.135:0/2604332436 >> 172.21.15.135:0/2604332436 conn(0x7f46500f09a0 msgr2=0x7f46500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:56.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 -- 172.21.15.135:0/2604332436 shutdown_connections 2023-11-15T06:39:56.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 -- 172.21.15.135:0/2604332436 wait complete. 2023-11-15T06:39:56.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.676+0000 7f4654ed4700 1 Processor -- start 2023-11-15T06:39:56.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.677+0000 7f4654ed4700 1 -- start start 2023-11-15T06:39:56.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.677+0000 7f4654ed4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46501053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:56.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.677+0000 7f4654ed4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46501058a0 con 0x7f46500f51a0 2023-11-15T06:39:56.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.677+0000 7f464f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46501053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:56.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.677+0000 7f464f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46501053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54648/0 (socket says 172.21.15.135:54648) 2023-11-15T06:39:56.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.677+0000 7f464f7fe700 1 -- 172.21.15.135:0/300518405 learned_addr learned my addr 172.21.15.135:0/300518405 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:56.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.678+0000 7f464f7fe700 1 -- 172.21.15.135:0/300518405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4640005040 con 0x7f46500f51a0 2023-11-15T06:39:56.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.678+0000 7f464f7fe700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46501053b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4640013510 tx=0x7f464000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:56.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.678+0000 7f4637fff700 1 -- 172.21.15.135:0/300518405 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4640009320 con 0x7f46500f51a0 2023-11-15T06:39:56.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.678+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4650105aa0 con 0x7f46500f51a0 2023-11-15T06:39:56.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.678+0000 7f4637fff700 1 -- 172.21.15.135:0/300518405 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4640013f00 con 0x7f46500f51a0 2023-11-15T06:39:56.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.678+0000 7f4637fff700 1 -- 172.21.15.135:0/300518405 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4640014dc0 con 0x7f46500f51a0 2023-11-15T06:39:56.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.678+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4650105f30 con 0x7f46500f51a0 2023-11-15T06:39:56.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.679+0000 7f4637fff700 1 -- 172.21.15.135:0/300518405 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4640014420 con 0x7f46500f51a0 2023-11-15T06:39:56.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.679+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4650045ad0 con 0x7f46500f51a0 2023-11-15T06:39:56.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.679+0000 7f4637fff700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f463802ff60 0x7f4638032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:56.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.679+0000 7f4637fff700 1 -- 172.21.15.135:0/300518405 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4640041110 con 0x7f46500f51a0 2023-11-15T06:39:56.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.679+0000 7f464effd700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f463802ff60 0x7f4638032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:56.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.679+0000 7f464effd700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f463802ff60 0x7f4638032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f46440096a0 tx=0x7f4644006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:56.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.684+0000 7f4637fff700 1 -- 172.21.15.135:0/300518405 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46400095f0 con 0x7f46500f51a0 2023-11-15T06:39:56.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.887+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4650106de0 con 0x7f46500f51a0 2023-11-15T06:39:56.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.888+0000 7f4637fff700 1 -- 172.21.15.135:0/300518405 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f464001cb10 con 0x7f46500f51a0 2023-11-15T06:39:56.891 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:56.892 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:56.892 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f463802ff60 msgr2=0x7f4638032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:56.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f463802ff60 0x7f4638032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f46440096a0 tx=0x7f4644006b40).stop 2023-11-15T06:39:56.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 msgr2=0x7f46501053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:56.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46501053b0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f4640013510 tx=0x7f464000b040).stop 2023-11-15T06:39:56.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 shutdown_connections 2023-11-15T06:39:56.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f463802ff60 0x7f4638032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:56.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 --2- 172.21.15.135:0/300518405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46500f51a0 0x7f46501053b0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:56.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 >> 172.21.15.135:0/300518405 conn(0x7f46500f09a0 msgr2=0x7f46500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:56.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 shutdown_connections 2023-11-15T06:39:56.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:56.890+0000 7f4654ed4700 1 -- 172.21.15.135:0/300518405 wait complete. 2023-11-15T06:39:56.895 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:39:57.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:56 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:56.889286+0000 mon.smithi123 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.135:0/300518405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:39:58.266 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:39:58.267 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:39:58.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:56.250717+0000 mgr.smithi123.razjtn (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:39:59.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.356+0000 7f6875f6e700 1 -- 172.21.15.135:0/1706781648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 msgr2=0x7f68700f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:59.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.356+0000 7f6875f6e700 1 --2- 172.21.15.135:0/1706781648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f68700f66c0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f6864009fb0 tx=0x7f6864009b30).stop 2023-11-15T06:39:59.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.356+0000 7f6875f6e700 1 -- 172.21.15.135:0/1706781648 shutdown_connections 2023-11-15T06:39:59.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.356+0000 7f6875f6e700 1 --2- 172.21.15.135:0/1706781648 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f68700f66c0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:59.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.356+0000 7f6875f6e700 1 -- 172.21.15.135:0/1706781648 >> 172.21.15.135:0/1706781648 conn(0x7f68700f1ac0 msgr2=0x7f68700f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:59.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.356+0000 7f6875f6e700 1 -- 172.21.15.135:0/1706781648 shutdown_connections 2023-11-15T06:39:59.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.356+0000 7f6875f6e700 1 -- 172.21.15.135:0/1706781648 wait complete. 2023-11-15T06:39:59.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.357+0000 7f6875f6e700 1 Processor -- start 2023-11-15T06:39:59.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.357+0000 7f6875f6e700 1 -- start start 2023-11-15T06:39:59.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.357+0000 7f6875f6e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f6870106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:59.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.357+0000 7f6875f6e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68701069f0 con 0x7f68700f62a0 2023-11-15T06:39:59.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.357+0000 7f6874f6c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f6870106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:59.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.357+0000 7f6874f6c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f6870106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58840/0 (socket says 172.21.15.135:58840) 2023-11-15T06:39:59.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.357+0000 7f6874f6c700 1 -- 172.21.15.135:0/1020345251 learned_addr learned my addr 172.21.15.135:0/1020345251 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:39:59.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.358+0000 7f6874f6c700 1 -- 172.21.15.135:0/1020345251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6864005040 con 0x7f68700f62a0 2023-11-15T06:39:59.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.358+0000 7f6874f6c700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f6870106500 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6864013510 tx=0x7f686400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:59.364 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.358+0000 7f686d7fa700 1 -- 172.21.15.135:0/1020345251 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6864009320 con 0x7f68700f62a0 2023-11-15T06:39:59.364 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.358+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6870106bf0 con 0x7f68700f62a0 2023-11-15T06:39:59.364 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.358+0000 7f686d7fa700 1 -- 172.21.15.135:0/1020345251 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6864013f00 con 0x7f68700f62a0 2023-11-15T06:39:59.364 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.358+0000 7f686d7fa700 1 -- 172.21.15.135:0/1020345251 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6864014dc0 con 0x7f68700f62a0 2023-11-15T06:39:59.364 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.358+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6870107080 con 0x7f68700f62a0 2023-11-15T06:39:59.365 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.359+0000 7f686d7fa700 1 -- 172.21.15.135:0/1020345251 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6864014420 con 0x7f68700f62a0 2023-11-15T06:39:59.366 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.359+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6870045ad0 con 0x7f68700f62a0 2023-11-15T06:39:59.366 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.359+0000 7f686d7fa700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f686002ff60 0x7f6860032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:39:59.366 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.359+0000 7f686d7fa700 1 -- 172.21.15.135:0/1020345251 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6864041110 con 0x7f68700f62a0 2023-11-15T06:39:59.366 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.359+0000 7f686ffff700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f686002ff60 0x7f6860032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:39:59.367 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.360+0000 7f686ffff700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f686002ff60 0x7f6860032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f685c0096a0 tx=0x7f685c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:39:59.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.364+0000 7f686d7fa700 1 -- 172.21.15.135:0/1020345251 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6864026bf0 con 0x7f68700f62a0 2023-11-15T06:39:59.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.570+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6870107f30 con 0x7f68700f62a0 2023-11-15T06:39:59.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.571+0000 7f686d7fa700 1 -- 172.21.15.135:0/1020345251 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f686401cb10 con 0x7f68700f62a0 2023-11-15T06:39:59.574 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:39:59.574 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:39:59.575 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f686002ff60 msgr2=0x7f6860032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:59.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f686002ff60 0x7f6860032420 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7f685c0096a0 tx=0x7f685c006b40).stop 2023-11-15T06:39:59.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 msgr2=0x7f6870106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:39:59.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f6870106500 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f6864013510 tx=0x7f686400b040).stop 2023-11-15T06:39:59.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 shutdown_connections 2023-11-15T06:39:59.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f686002ff60 0x7f6860032420 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:59.577 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 --2- 172.21.15.135:0/1020345251 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68700f62a0 0x7f6870106500 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:39:59.577 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 >> 172.21.15.135:0/1020345251 conn(0x7f68700f1ac0 msgr2=0x7f68700f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:39:59.577 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 shutdown_connections 2023-11-15T06:39:59.577 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:39:59.573+0000 7f6875f6e700 1 -- 172.21.15.135:0/1020345251 wait complete. 2023-11-15T06:39:59.578 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:00.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:39:58.251109+0000 mgr.smithi123.razjtn (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:00.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:39:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:39:59.572015+0000 mon.smithi123 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.135:0/1020345251' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:00.947 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:00.948 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:02.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.107+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1503867999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 msgr2=0x7fdadc0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:02.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.107+0000 7fdae1d9a700 1 --2- 172.21.15.135:0/1503867999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc0f9190 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fdac4009fb0 tx=0x7fdac4009b30).stop 2023-11-15T06:40:02.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.107+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1503867999 shutdown_connections 2023-11-15T06:40:02.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.107+0000 7fdae1d9a700 1 --2- 172.21.15.135:0/1503867999 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc0f9190 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:02.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.107+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1503867999 >> 172.21.15.135:0/1503867999 conn(0x7fdadc0f0b70 msgr2=0x7fdadc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:02.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.107+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1503867999 shutdown_connections 2023-11-15T06:40:02.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.107+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1503867999 wait complete. 2023-11-15T06:40:02.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.108+0000 7fdae1d9a700 1 Processor -- start 2023-11-15T06:40:02.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.108+0000 7fdae1d9a700 1 -- start start 2023-11-15T06:40:02.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.108+0000 7fdae1d9a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:02.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.108+0000 7fdae1d9a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdadc106bd0 con 0x7fdadc0601e0 2023-11-15T06:40:02.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.108+0000 7fdadb7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:02.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.108+0000 7fdadb7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58854/0 (socket says 172.21.15.135:58854) 2023-11-15T06:40:02.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.108+0000 7fdadb7fe700 1 -- 172.21.15.135:0/1265114990 learned_addr learned my addr 172.21.15.135:0/1265114990 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:02.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.109+0000 7fdadb7fe700 1 -- 172.21.15.135:0/1265114990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdac4005040 con 0x7fdadc0601e0 2023-11-15T06:40:02.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.109+0000 7fdadb7fe700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc1066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdac4004dd0 tx=0x7fdac4009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:02.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.109+0000 7fdad3fff700 1 -- 172.21.15.135:0/1265114990 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac4018070 con 0x7fdadc0601e0 2023-11-15T06:40:02.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.109+0000 7fdad3fff700 1 -- 172.21.15.135:0/1265114990 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdac4013410 con 0x7fdadc0601e0 2023-11-15T06:40:02.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.109+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdadc106dd0 con 0x7fdadc0601e0 2023-11-15T06:40:02.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.109+0000 7fdad3fff700 1 -- 172.21.15.135:0/1265114990 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdac401c3c0 con 0x7fdadc0601e0 2023-11-15T06:40:02.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.109+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdadc107260 con 0x7fdadc0601e0 2023-11-15T06:40:02.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.110+0000 7fdad3fff700 1 -- 172.21.15.135:0/1265114990 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdac401d420 con 0x7fdadc0601e0 2023-11-15T06:40:02.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.110+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdadc045ad0 con 0x7fdadc0601e0 2023-11-15T06:40:02.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.110+0000 7fdad3fff700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdac802ff60 0x7fdac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:02.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.110+0000 7fdad3fff700 1 -- 172.21.15.135:0/1265114990 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdac4041510 con 0x7fdadc0601e0 2023-11-15T06:40:02.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.110+0000 7fdadaffd700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdac802ff60 0x7fdac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:02.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.111+0000 7fdadaffd700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdac802ff60 0x7fdac8032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdacc0096a0 tx=0x7fdacc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:02.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.114+0000 7fdad3fff700 1 -- 172.21.15.135:0/1265114990 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdac401d820 con 0x7fdadc0601e0 2023-11-15T06:40:02.326 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.322+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdadc108110 con 0x7fdadc0601e0 2023-11-15T06:40:02.326 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.323+0000 7fdad3fff700 1 -- 172.21.15.135:0/1265114990 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdac401b370 con 0x7fdadc0601e0 2023-11-15T06:40:02.326 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:02.327 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:02.327 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdac802ff60 msgr2=0x7fdac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:02.328 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdac802ff60 0x7fdac8032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fdacc0096a0 tx=0x7fdacc006b40).stop 2023-11-15T06:40:02.328 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 msgr2=0x7fdadc1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:02.328 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc1066e0 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fdac4004dd0 tx=0x7fdac4009b30).stop 2023-11-15T06:40:02.328 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 shutdown_connections 2023-11-15T06:40:02.328 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdac802ff60 0x7fdac8032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:02.329 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 --2- 172.21.15.135:0/1265114990 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdadc0601e0 0x7fdadc1066e0 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:02.329 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.324+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 >> 172.21.15.135:0/1265114990 conn(0x7fdadc0f0b70 msgr2=0x7fdadc0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:02.329 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.325+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 shutdown_connections 2023-11-15T06:40:02.329 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:02.325+0000 7fdae1d9a700 1 -- 172.21.15.135:0/1265114990 wait complete. 2023-11-15T06:40:02.329 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:02.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:00.251376+0000 mgr.smithi123.razjtn (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:03.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:02 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:02.323865+0000 mon.smithi123 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.135:0/1265114990' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:03.701 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:03.702 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:04.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:02.251709+0000 mgr.smithi123.razjtn (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:04.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.854+0000 7f29a3c09700 1 -- 172.21.15.135:0/1964945049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 msgr2=0x7f299c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:04.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.854+0000 7f29a3c09700 1 --2- 172.21.15.135:0/1964945049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c0f6930 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f298c009fb0 tx=0x7f298c009b30).stop 2023-11-15T06:40:04.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.854+0000 7f29a3c09700 1 -- 172.21.15.135:0/1964945049 shutdown_connections 2023-11-15T06:40:04.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.854+0000 7f29a3c09700 1 --2- 172.21.15.135:0/1964945049 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c0f6930 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:04.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.854+0000 7f29a3c09700 1 -- 172.21.15.135:0/1964945049 >> 172.21.15.135:0/1964945049 conn(0x7f299c0f1c70 msgr2=0x7f299c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:04.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.854+0000 7f29a3c09700 1 -- 172.21.15.135:0/1964945049 shutdown_connections 2023-11-15T06:40:04.859 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.854+0000 7f29a3c09700 1 -- 172.21.15.135:0/1964945049 wait complete. 2023-11-15T06:40:04.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a3c09700 1 Processor -- start 2023-11-15T06:40:04.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a3c09700 1 -- start start 2023-11-15T06:40:04.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a3c09700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:04.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a3c09700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f299c106c20 con 0x7f299c0f6510 2023-11-15T06:40:04.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a19a5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:04.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a19a5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58886/0 (socket says 172.21.15.135:58886) 2023-11-15T06:40:04.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a19a5700 1 -- 172.21.15.135:0/572746438 learned_addr learned my addr 172.21.15.135:0/572746438 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:04.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.855+0000 7f29a19a5700 1 -- 172.21.15.135:0/572746438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f298c005040 con 0x7f299c0f6510 2023-11-15T06:40:04.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.856+0000 7f29a19a5700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f298c013510 tx=0x7f298c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:04.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.856+0000 7f29927fc700 1 -- 172.21.15.135:0/572746438 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f298c009320 con 0x7f299c0f6510 2023-11-15T06:40:04.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.856+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f299c106e20 con 0x7f299c0f6510 2023-11-15T06:40:04.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.856+0000 7f29927fc700 1 -- 172.21.15.135:0/572746438 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f298c013f00 con 0x7f299c0f6510 2023-11-15T06:40:04.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.856+0000 7f29927fc700 1 -- 172.21.15.135:0/572746438 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f298c014d90 con 0x7f299c0f6510 2023-11-15T06:40:04.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.856+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f299c1072b0 con 0x7f299c0f6510 2023-11-15T06:40:04.864 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.857+0000 7f29927fc700 1 -- 172.21.15.135:0/572746438 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f298c0263a0 con 0x7f299c0f6510 2023-11-15T06:40:04.864 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.857+0000 7f29927fc700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f298802ff60 0x7f2988032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:04.864 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.857+0000 7f29927fc700 1 -- 172.21.15.135:0/572746438 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f298c040fe0 con 0x7f299c0f6510 2023-11-15T06:40:04.864 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.857+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f299c045ad0 con 0x7f299c0f6510 2023-11-15T06:40:04.865 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.857+0000 7f29a11a4700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f298802ff60 0x7f2988032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:04.865 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.858+0000 7f29a11a4700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f298802ff60 0x7f2988032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f29980096a0 tx=0x7f2998006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:04.865 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:04.862+0000 7f29927fc700 1 -- 172.21.15.135:0/572746438 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f298c012920 con 0x7f299c0f6510 2023-11-15T06:40:05.068 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.066+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f299c0fc050 con 0x7f299c0f6510 2023-11-15T06:40:05.069 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.066+0000 7f29927fc700 1 -- 172.21.15.135:0/572746438 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f298c02b020 con 0x7f299c0f6510 2023-11-15T06:40:05.069 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:05.070 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:05.071 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f298802ff60 msgr2=0x7f2988032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:05.071 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f298802ff60 0x7f2988032420 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f29980096a0 tx=0x7f2998006b40).stop 2023-11-15T06:40:05.071 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 msgr2=0x7f299c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:05.071 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c106730 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f298c013510 tx=0x7f298c00b040).stop 2023-11-15T06:40:05.072 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 shutdown_connections 2023-11-15T06:40:05.072 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f298802ff60 0x7f2988032420 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:05.072 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 --2- 172.21.15.135:0/572746438 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f299c0f6510 0x7f299c106730 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:05.072 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 >> 172.21.15.135:0/572746438 conn(0x7f299c0f1c70 msgr2=0x7f299c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:05.072 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 shutdown_connections 2023-11-15T06:40:05.073 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:05.068+0000 7f29a3c09700 1 -- 172.21.15.135:0/572746438 wait complete. 2023-11-15T06:40:05.073 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:06.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:04.252094+0000 mgr.smithi123.razjtn (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:06.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:05.067386+0000 mon.smithi123 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.135:0/572746438' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:06.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:06.451 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:07.551 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.547+0000 7f66b3b87700 1 -- 172.21.15.135:0/2771146785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 msgr2=0x7f66ac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:07.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.547+0000 7f66b3b87700 1 --2- 172.21.15.135:0/2771146785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac0f90f0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f669c009fb0 tx=0x7f669c009b30).stop 2023-11-15T06:40:07.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.547+0000 7f66b3b87700 1 -- 172.21.15.135:0/2771146785 shutdown_connections 2023-11-15T06:40:07.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.547+0000 7f66b3b87700 1 --2- 172.21.15.135:0/2771146785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac0f90f0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:07.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.547+0000 7f66b3b87700 1 -- 172.21.15.135:0/2771146785 >> 172.21.15.135:0/2771146785 conn(0x7f66ac0f0b70 msgr2=0x7f66ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:07.552 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.547+0000 7f66b3b87700 1 -- 172.21.15.135:0/2771146785 shutdown_connections 2023-11-15T06:40:07.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.547+0000 7f66b3b87700 1 -- 172.21.15.135:0/2771146785 wait complete. 2023-11-15T06:40:07.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b3b87700 1 Processor -- start 2023-11-15T06:40:07.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b3b87700 1 -- start start 2023-11-15T06:40:07.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b3b87700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac1088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:07.553 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b3b87700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66ac108de0 con 0x7f66ac060280 2023-11-15T06:40:07.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b1923700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac1088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:07.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b1923700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac1088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58894/0 (socket says 172.21.15.135:58894) 2023-11-15T06:40:07.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b1923700 1 -- 172.21.15.135:0/4285466051 learned_addr learned my addr 172.21.15.135:0/4285466051 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:07.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b1923700 1 -- 172.21.15.135:0/4285466051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f669c005040 con 0x7f66ac060280 2023-11-15T06:40:07.554 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.548+0000 7f66b1923700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac1088f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f669c013510 tx=0x7f669c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:07.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.549+0000 7f66a27fc700 1 -- 172.21.15.135:0/4285466051 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669c009320 con 0x7f66ac060280 2023-11-15T06:40:07.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.549+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66ac108fe0 con 0x7f66ac060280 2023-11-15T06:40:07.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.549+0000 7f66a27fc700 1 -- 172.21.15.135:0/4285466051 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f669c013f00 con 0x7f66ac060280 2023-11-15T06:40:07.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.549+0000 7f66a27fc700 1 -- 172.21.15.135:0/4285466051 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f669c014dc0 con 0x7f66ac060280 2023-11-15T06:40:07.555 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.549+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66ac109470 con 0x7f66ac060280 2023-11-15T06:40:07.556 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.549+0000 7f66a27fc700 1 -- 172.21.15.135:0/4285466051 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f669c01d3d0 con 0x7f66ac060280 2023-11-15T06:40:07.557 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.550+0000 7f66a27fc700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f669802ff60 0x7f6698032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:07.557 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.550+0000 7f66a27fc700 1 -- 172.21.15.135:0/4285466051 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f669c041100 con 0x7f66ac060280 2023-11-15T06:40:07.557 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.550+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66ac045ad0 con 0x7f66ac060280 2023-11-15T06:40:07.557 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.550+0000 7f66b1122700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f669802ff60 0x7f6698032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:07.557 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.550+0000 7f66b1122700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f669802ff60 0x7f6698032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f66a80096a0 tx=0x7f66a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:07.558 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.554+0000 7f66a27fc700 1 -- 172.21.15.135:0/4285466051 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f669c012900 con 0x7f66ac060280 2023-11-15T06:40:07.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.763+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66ac10a190 con 0x7f66ac060280 2023-11-15T06:40:07.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.764+0000 7f66a27fc700 1 -- 172.21.15.135:0/4285466051 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f669c02b020 con 0x7f66ac060280 2023-11-15T06:40:07.767 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:07.767 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:07.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f669802ff60 msgr2=0x7f6698032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:07.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f669802ff60 0x7f6698032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f66a80096a0 tx=0x7f66a8006b40).stop 2023-11-15T06:40:07.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 msgr2=0x7f66ac1088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:07.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac1088f0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f669c013510 tx=0x7f669c00b040).stop 2023-11-15T06:40:07.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 shutdown_connections 2023-11-15T06:40:07.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f669802ff60 0x7f6698032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:07.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 --2- 172.21.15.135:0/4285466051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66ac060280 0x7f66ac1088f0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:07.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 >> 172.21.15.135:0/4285466051 conn(0x7f66ac0f0b70 msgr2=0x7f66ac0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:07.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 shutdown_connections 2023-11-15T06:40:07.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:07.766+0000 7f66b3b87700 1 -- 172.21.15.135:0/4285466051 wait complete. 2023-11-15T06:40:07.771 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:08.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:06.252473+0000 mgr.smithi123.razjtn (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:08.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:07.765238+0000 mon.smithi123 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.135:0/4285466051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:09.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:09.104 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:10.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/399200816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 msgr2=0x7fbbb00f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:10.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 --2- 172.21.15.135:0/399200816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f8f80 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fbba0009fb0 tx=0x7fbba0009b30).stop 2023-11-15T06:40:10.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/399200816 shutdown_connections 2023-11-15T06:40:10.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 --2- 172.21.15.135:0/399200816 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f8f80 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:10.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/399200816 >> 172.21.15.135:0/399200816 conn(0x7fbbb00f09a0 msgr2=0x7fbbb00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:10.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/399200816 shutdown_connections 2023-11-15T06:40:10.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/399200816 wait complete. 2023-11-15T06:40:10.199 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 Processor -- start 2023-11-15T06:40:10.199 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 -- start start 2023-11-15T06:40:10.199 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:10.199 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.194+0000 7fbbb4b0c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbbb00f4e80 con 0x7fbbb0060180 2023-11-15T06:40:10.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbbaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:10.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbbaf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44830/0 (socket says 172.21.15.135:44830) 2023-11-15T06:40:10.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbbaf7fe700 1 -- 172.21.15.135:0/3041209585 learned_addr learned my addr 172.21.15.135:0/3041209585 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:10.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbbaf7fe700 1 -- 172.21.15.135:0/3041209585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbba0005040 con 0x7fbbb0060180 2023-11-15T06:40:10.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbbaf7fe700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f6a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbba0013510 tx=0x7fbba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:10.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbb97fff700 1 -- 172.21.15.135:0/3041209585 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba0009320 con 0x7fbbb0060180 2023-11-15T06:40:10.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbbb00f5020 con 0x7fbbb0060180 2023-11-15T06:40:10.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbb97fff700 1 -- 172.21.15.135:0/3041209585 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbba0013f00 con 0x7fbbb0060180 2023-11-15T06:40:10.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.195+0000 7fbb97fff700 1 -- 172.21.15.135:0/3041209585 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbba0014d90 con 0x7fbbb0060180 2023-11-15T06:40:10.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.196+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbbb00f54b0 con 0x7fbbb0060180 2023-11-15T06:40:10.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.196+0000 7fbb97fff700 1 -- 172.21.15.135:0/3041209585 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbba00263a0 con 0x7fbbb0060180 2023-11-15T06:40:10.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.196+0000 7fbb97fff700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb9802ff60 0x7fbb98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:10.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.196+0000 7fbb97fff700 1 -- 172.21.15.135:0/3041209585 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbba0041020 con 0x7fbbb0060180 2023-11-15T06:40:10.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.196+0000 7fbbaeffd700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb9802ff60 0x7fbb98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:10.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.197+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbbb0059100 con 0x7fbbb0060180 2023-11-15T06:40:10.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.197+0000 7fbbaeffd700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb9802ff60 0x7fbb98032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbba40096a0 tx=0x7fbba4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:10.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.201+0000 7fbb97fff700 1 -- 172.21.15.135:0/3041209585 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbba0012920 con 0x7fbbb0060180 2023-11-15T06:40:10.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:08.252849+0000 mgr.smithi123.razjtn (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:10.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.410+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbbb00f5b10 con 0x7fbbb0060180 2023-11-15T06:40:10.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.411+0000 7fbb97fff700 1 -- 172.21.15.135:0/3041209585 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbba002b020 con 0x7fbbb0060180 2023-11-15T06:40:10.414 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:10.414 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:10.415 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb9802ff60 msgr2=0x7fbb98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:10.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb9802ff60 0x7fbb98032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fbba40096a0 tx=0x7fbba4006b40).stop 2023-11-15T06:40:10.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 msgr2=0x7fbbb00f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:10.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f6a00 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fbba0013510 tx=0x7fbba000b040).stop 2023-11-15T06:40:10.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 shutdown_connections 2023-11-15T06:40:10.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb9802ff60 0x7fbb98032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:10.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 --2- 172.21.15.135:0/3041209585 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbbb0060180 0x7fbbb00f6a00 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:10.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 >> 172.21.15.135:0/3041209585 conn(0x7fbbb00f09a0 msgr2=0x7fbbb00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:10.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 shutdown_connections 2023-11-15T06:40:10.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:10.413+0000 7fbbb4b0c700 1 -- 172.21.15.135:0/3041209585 wait complete. 2023-11-15T06:40:10.419 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:11.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:10 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:10.412167+0000 mon.smithi123 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.135:0/3041209585' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:11.796 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:11.797 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:12.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:10.253072+0000 mgr.smithi123.razjtn (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:12.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.926+0000 7f0d762fe700 1 -- 172.21.15.135:0/1882180311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 msgr2=0x7f0d700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:12.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.926+0000 7f0d762fe700 1 --2- 172.21.15.135:0/1882180311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d700f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f0d60009fb0 tx=0x7f0d60009b30).stop 2023-11-15T06:40:12.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.926+0000 7f0d762fe700 1 -- 172.21.15.135:0/1882180311 shutdown_connections 2023-11-15T06:40:12.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.926+0000 7f0d762fe700 1 --2- 172.21.15.135:0/1882180311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d700f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:12.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.926+0000 7f0d762fe700 1 -- 172.21.15.135:0/1882180311 >> 172.21.15.135:0/1882180311 conn(0x7f0d700f1c70 msgr2=0x7f0d700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:12.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.926+0000 7f0d762fe700 1 -- 172.21.15.135:0/1882180311 shutdown_connections 2023-11-15T06:40:12.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.926+0000 7f0d762fe700 1 -- 172.21.15.135:0/1882180311 wait complete. 2023-11-15T06:40:12.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.927+0000 7f0d762fe700 1 Processor -- start 2023-11-15T06:40:12.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.927+0000 7f0d762fe700 1 -- start start 2023-11-15T06:40:12.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.927+0000 7f0d762fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d7006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:12.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.927+0000 7f0d762fe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0d70069e20 con 0x7f0d700f6510 2023-11-15T06:40:12.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.927+0000 7f0d6ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d7006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:12.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.927+0000 7f0d6ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d7006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44858/0 (socket says 172.21.15.135:44858) 2023-11-15T06:40:12.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.927+0000 7f0d6ffff700 1 -- 172.21.15.135:0/872832017 learned_addr learned my addr 172.21.15.135:0/872832017 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:12.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.928+0000 7f0d6ffff700 1 -- 172.21.15.135:0/872832017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0d60005040 con 0x7f0d700f6510 2023-11-15T06:40:12.934 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.928+0000 7f0d6ffff700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d7006b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0d60014000 tx=0x7f0d6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:12.934 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.928+0000 7f0d6cff9700 1 -- 172.21.15.135:0/872832017 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d600095f0 con 0x7f0d700f6510 2023-11-15T06:40:12.934 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.928+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0d70069fc0 con 0x7f0d700f6510 2023-11-15T06:40:12.934 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.928+0000 7f0d6cff9700 1 -- 172.21.15.135:0/872832017 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0d60007e70 con 0x7f0d700f6510 2023-11-15T06:40:12.934 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.928+0000 7f0d6cff9700 1 -- 172.21.15.135:0/872832017 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0d60014ea0 con 0x7f0d700f6510 2023-11-15T06:40:12.935 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.928+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0d7006a450 con 0x7f0d700f6510 2023-11-15T06:40:12.936 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.929+0000 7f0d6cff9700 1 -- 172.21.15.135:0/872832017 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0d60025070 con 0x7f0d700f6510 2023-11-15T06:40:12.936 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.929+0000 7f0d6cff9700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:12.936 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.929+0000 7f0d6cff9700 1 -- 172.21.15.135:0/872832017 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0d60040980 con 0x7f0d700f6510 2023-11-15T06:40:12.936 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.929+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0d70059100 con 0x7f0d700f6510 2023-11-15T06:40:12.936 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.929+0000 7f0d6f7fe700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:12.937 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.930+0000 7f0d6f7fe700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d5802ff60 0x7f0d58032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0d640096a0 tx=0x7f0d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:12.937 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:12.934+0000 7f0d6cff9700 1 -- 172.21.15.135:0/872832017 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0d60026350 con 0x7f0d700f6510 2023-11-15T06:40:13.142 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.139+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0d70045ad0 con 0x7f0d700f6510 2023-11-15T06:40:13.143 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.140+0000 7f0d6cff9700 1 -- 172.21.15.135:0/872832017 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0d60012330 con 0x7f0d700f6510 2023-11-15T06:40:13.143 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:13.144 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:13.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.141+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d5802ff60 msgr2=0x7f0d58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:13.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.141+0000 7f0d762fe700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d5802ff60 0x7f0d58032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f0d640096a0 tx=0x7f0d64006b40).stop 2023-11-15T06:40:13.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.141+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 msgr2=0x7f0d7006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:13.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.141+0000 7f0d762fe700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d7006b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f0d60014000 tx=0x7f0d6000b040).stop 2023-11-15T06:40:13.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.142+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 shutdown_connections 2023-11-15T06:40:13.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.142+0000 7f0d762fe700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f0d5802ff60 0x7f0d58032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:13.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.142+0000 7f0d762fe700 1 --2- 172.21.15.135:0/872832017 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f0d700f6510 0x7f0d7006b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:13.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.142+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 >> 172.21.15.135:0/872832017 conn(0x7f0d700f1c70 msgr2=0x7f0d700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:13.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.142+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 shutdown_connections 2023-11-15T06:40:13.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:13.142+0000 7f0d762fe700 1 -- 172.21.15.135:0/872832017 wait complete. 2023-11-15T06:40:13.148 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:14.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:12.253434+0000 mgr.smithi123.razjtn (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:14.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:13.140864+0000 mon.smithi123 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.135:0/872832017' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:14.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:14.532 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:15.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.702+0000 7f08b18e2700 1 -- 172.21.15.135:0/3812760608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 msgr2=0x7f08ac0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:15.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.702+0000 7f08b18e2700 1 --2- 172.21.15.135:0/3812760608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac0f6930 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f08a0009fb0 tx=0x7f08a0009b30).stop 2023-11-15T06:40:15.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.702+0000 7f08b18e2700 1 -- 172.21.15.135:0/3812760608 shutdown_connections 2023-11-15T06:40:15.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.702+0000 7f08b18e2700 1 --2- 172.21.15.135:0/3812760608 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac0f6930 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:15.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.702+0000 7f08b18e2700 1 -- 172.21.15.135:0/3812760608 >> 172.21.15.135:0/3812760608 conn(0x7f08ac0f1c70 msgr2=0x7f08ac0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:15.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.702+0000 7f08b18e2700 1 -- 172.21.15.135:0/3812760608 shutdown_connections 2023-11-15T06:40:15.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.702+0000 7f08b18e2700 1 -- 172.21.15.135:0/3812760608 wait complete. 2023-11-15T06:40:15.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.703+0000 7f08b18e2700 1 Processor -- start 2023-11-15T06:40:15.709 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.703+0000 7f08b18e2700 1 -- start start 2023-11-15T06:40:15.709 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.703+0000 7f08b18e2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:15.709 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.703+0000 7f08b18e2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f08ac106c20 con 0x7f08ac0f6510 2023-11-15T06:40:15.709 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.703+0000 7f08aaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:15.709 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.703+0000 7f08aaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44880/0 (socket says 172.21.15.135:44880) 2023-11-15T06:40:15.710 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.703+0000 7f08aaffd700 1 -- 172.21.15.135:0/3517469758 learned_addr learned my addr 172.21.15.135:0/3517469758 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:15.710 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.704+0000 7f08aaffd700 1 -- 172.21.15.135:0/3517469758 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08a0005040 con 0x7f08ac0f6510 2023-11-15T06:40:15.710 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.704+0000 7f08aaffd700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f08a0014000 tx=0x7f08a000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:15.710 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.704+0000 7f08937fe700 1 -- 172.21.15.135:0/3517469758 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08a00095f0 con 0x7f08ac0f6510 2023-11-15T06:40:15.710 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.704+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f08ac106e20 con 0x7f08ac0f6510 2023-11-15T06:40:15.711 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.704+0000 7f08937fe700 1 -- 172.21.15.135:0/3517469758 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08a0007e70 con 0x7f08ac0f6510 2023-11-15T06:40:15.711 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.704+0000 7f08937fe700 1 -- 172.21.15.135:0/3517469758 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08a0014ea0 con 0x7f08ac0f6510 2023-11-15T06:40:15.711 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.704+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f08ac1072b0 con 0x7f08ac0f6510 2023-11-15T06:40:15.712 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.705+0000 7f08937fe700 1 -- 172.21.15.135:0/3517469758 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08a0025070 con 0x7f08ac0f6510 2023-11-15T06:40:15.712 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.705+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f08ac045ad0 con 0x7f08ac0f6510 2023-11-15T06:40:15.712 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.705+0000 7f08937fe700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f089402ff60 0x7f0894032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:15.713 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.705+0000 7f08937fe700 1 -- 172.21.15.135:0/3517469758 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f08a0040920 con 0x7f08ac0f6510 2023-11-15T06:40:15.713 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.705+0000 7f08aa7fc700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f089402ff60 0x7f0894032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:15.713 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.706+0000 7f08aa7fc700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f089402ff60 0x7f0894032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f089c0096a0 tx=0x7f089c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:15.713 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.710+0000 7f08937fe700 1 -- 172.21.15.135:0/3517469758 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08a000f4e0 con 0x7f08ac0f6510 2023-11-15T06:40:15.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.916+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f08ac0fbf60 con 0x7f08ac0f6510 2023-11-15T06:40:15.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.917+0000 7f08937fe700 1 -- 172.21.15.135:0/3517469758 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f08a000f300 con 0x7f08ac0f6510 2023-11-15T06:40:15.920 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:15.920 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:15.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f089402ff60 msgr2=0x7f0894032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:15.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f089402ff60 0x7f0894032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f089c0096a0 tx=0x7f089c006b40).stop 2023-11-15T06:40:15.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 msgr2=0x7f08ac106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:15.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac106730 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f08a0014000 tx=0x7f08a000b040).stop 2023-11-15T06:40:15.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 shutdown_connections 2023-11-15T06:40:15.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f089402ff60 0x7f0894032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:15.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 --2- 172.21.15.135:0/3517469758 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f08ac0f6510 0x7f08ac106730 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:15.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 >> 172.21.15.135:0/3517469758 conn(0x7f08ac0f1c70 msgr2=0x7f08ac0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:15.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 shutdown_connections 2023-11-15T06:40:15.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:15.919+0000 7f08b18e2700 1 -- 172.21.15.135:0/3517469758 wait complete. 2023-11-15T06:40:15.924 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:16.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:14.253815+0000 mgr.smithi123.razjtn (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:16.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:15.918136+0000 mon.smithi123 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.135:0/3517469758' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:17.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:17.315 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:18.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:16.254044+0000 mgr.smithi123.razjtn (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:18.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.458+0000 7f9373460700 1 -- 172.21.15.135:0/4024749866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 msgr2=0x7f936c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:18.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.458+0000 7f9373460700 1 --2- 172.21.15.135:0/4024749866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c0f55c0 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7f935c009fb0 tx=0x7f935c009b30).stop 2023-11-15T06:40:18.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.459+0000 7f9373460700 1 -- 172.21.15.135:0/4024749866 shutdown_connections 2023-11-15T06:40:18.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.459+0000 7f9373460700 1 --2- 172.21.15.135:0/4024749866 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c0f55c0 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:18.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.459+0000 7f9373460700 1 -- 172.21.15.135:0/4024749866 >> 172.21.15.135:0/4024749866 conn(0x7f936c0f09a0 msgr2=0x7f936c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:18.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.459+0000 7f9373460700 1 -- 172.21.15.135:0/4024749866 shutdown_connections 2023-11-15T06:40:18.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.459+0000 7f9373460700 1 -- 172.21.15.135:0/4024749866 wait complete. 2023-11-15T06:40:18.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.459+0000 7f9373460700 1 Processor -- start 2023-11-15T06:40:18.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.459+0000 7f9373460700 1 -- start start 2023-11-15T06:40:18.465 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f9373460700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:18.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f9373460700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f936c060220 con 0x7f936c0f51a0 2023-11-15T06:40:18.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f937245e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:18.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f937245e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57428/0 (socket says 172.21.15.135:57428) 2023-11-15T06:40:18.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f937245e700 1 -- 172.21.15.135:0/2643392425 learned_addr learned my addr 172.21.15.135:0/2643392425 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:18.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f937245e700 1 -- 172.21.15.135:0/2643392425 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f935c005040 con 0x7f936c0f51a0 2023-11-15T06:40:18.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f937245e700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c1091d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f935c013510 tx=0x7f935c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:18.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f9362ffd700 1 -- 172.21.15.135:0/2643392425 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c009320 con 0x7f936c0f51a0 2023-11-15T06:40:18.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f936c060420 con 0x7f936c0f51a0 2023-11-15T06:40:18.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f9362ffd700 1 -- 172.21.15.135:0/2643392425 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f935c013f00 con 0x7f936c0f51a0 2023-11-15T06:40:18.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f9362ffd700 1 -- 172.21.15.135:0/2643392425 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f935c014dc0 con 0x7f936c0f51a0 2023-11-15T06:40:18.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.460+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f936c109880 con 0x7f936c0f51a0 2023-11-15T06:40:18.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.461+0000 7f9362ffd700 1 -- 172.21.15.135:0/2643392425 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f935c0095f0 con 0x7f936c0f51a0 2023-11-15T06:40:18.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.461+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f936c059100 con 0x7f936c0f51a0 2023-11-15T06:40:18.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.461+0000 7f9362ffd700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f935802ff60 0x7f9358032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:18.470 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.461+0000 7f9362ffd700 1 -- 172.21.15.135:0/2643392425 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f935c041020 con 0x7f936c0f51a0 2023-11-15T06:40:18.470 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.462+0000 7f9371c5d700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f935802ff60 0x7f9358032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:18.470 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.462+0000 7f9371c5d700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f935802ff60 0x7f9358032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f93680096a0 tx=0x7f9368006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:18.471 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.466+0000 7f9362ffd700 1 -- 172.21.15.135:0/2643392425 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f935c014420 con 0x7f936c0f51a0 2023-11-15T06:40:18.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.675+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f936c045ad0 con 0x7f936c0f51a0 2023-11-15T06:40:18.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.675+0000 7f9362ffd700 1 -- 172.21.15.135:0/2643392425 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f935c01c920 con 0x7f936c0f51a0 2023-11-15T06:40:18.678 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:18.679 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:18.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f935802ff60 msgr2=0x7f9358032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:18.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f935802ff60 0x7f9358032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f93680096a0 tx=0x7f9368006b40).stop 2023-11-15T06:40:18.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 msgr2=0x7f936c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:18.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c1091d0 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f935c013510 tx=0x7f935c00b040).stop 2023-11-15T06:40:18.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 shutdown_connections 2023-11-15T06:40:18.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f935802ff60 0x7f9358032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:18.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 --2- 172.21.15.135:0/2643392425 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f936c0f51a0 0x7f936c1091d0 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:18.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 >> 172.21.15.135:0/2643392425 conn(0x7f936c0f09a0 msgr2=0x7f936c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:18.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 shutdown_connections 2023-11-15T06:40:18.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:18.677+0000 7f9373460700 1 -- 172.21.15.135:0/2643392425 wait complete. 2023-11-15T06:40:18.682 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:19.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:18 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:18.676402+0000 mon.smithi123 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.135:0/2643392425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:20.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:20.066 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:20.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:18.254385+0000 mgr.smithi123.razjtn (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:21.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 -- 172.21.15.135:0/3133054230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 msgr2=0x7f86d80f4c00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:21.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 --2- 172.21.15.135:0/3133054230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f4c00 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f86d0009960 tx=0x7f86d0004d10).stop 2023-11-15T06:40:21.283 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 -- 172.21.15.135:0/3133054230 shutdown_connections 2023-11-15T06:40:21.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 --2- 172.21.15.135:0/3133054230 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f4c00 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:21.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 -- 172.21.15.135:0/3133054230 >> 172.21.15.135:0/3133054230 conn(0x7f86d80f0260 msgr2=0x7f86d80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:21.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 -- 172.21.15.135:0/3133054230 shutdown_connections 2023-11-15T06:40:21.284 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 -- 172.21.15.135:0/3133054230 wait complete. 2023-11-15T06:40:21.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.280+0000 7f86dd250700 1 Processor -- start 2023-11-15T06:40:21.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.281+0000 7f86dd250700 1 -- start start 2023-11-15T06:40:21.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.281+0000 7f86dd250700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f73d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:21.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.281+0000 7f86dd250700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f86d80f5850 con 0x7f86d80f47e0 2023-11-15T06:40:21.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.281+0000 7f86d6d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f73d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:21.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.281+0000 7f86d6d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f73d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57448/0 (socket says 172.21.15.135:57448) 2023-11-15T06:40:21.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.281+0000 7f86d6d9d700 1 -- 172.21.15.135:0/864257148 learned_addr learned my addr 172.21.15.135:0/864257148 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:21.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.281+0000 7f86d6d9d700 1 -- 172.21.15.135:0/864257148 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f86d0009640 con 0x7f86d80f47e0 2023-11-15T06:40:21.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.282+0000 7f86d6d9d700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f73d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f86d0007bd0 tx=0x7f86d0004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:21.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.282+0000 7f86cf7fe700 1 -- 172.21.15.135:0/864257148 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d0018070 con 0x7f86d80f47e0 2023-11-15T06:40:21.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.282+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f86d80f59f0 con 0x7f86d80f47e0 2023-11-15T06:40:21.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.282+0000 7f86cf7fe700 1 -- 172.21.15.135:0/864257148 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f86d000b870 con 0x7f86d80f47e0 2023-11-15T06:40:21.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.282+0000 7f86cf7fe700 1 -- 172.21.15.135:0/864257148 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86d001d420 con 0x7f86d80f47e0 2023-11-15T06:40:21.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.282+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86d80f5e80 con 0x7f86d80f47e0 2023-11-15T06:40:21.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.283+0000 7f86cf7fe700 1 -- 172.21.15.135:0/864257148 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f86d0024060 con 0x7f86d80f47e0 2023-11-15T06:40:21.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.283+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f86d8059100 con 0x7f86d80f47e0 2023-11-15T06:40:21.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.283+0000 7f86cf7fe700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f86c402ffb0 0x7f86c4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:21.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.283+0000 7f86cf7fe700 1 -- 172.21.15.135:0/864257148 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86d00411b0 con 0x7f86d80f47e0 2023-11-15T06:40:21.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.283+0000 7f86d659c700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f86c402ffb0 0x7f86c4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:21.292 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.283+0000 7f86d659c700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f86c402ffb0 0x7f86c4032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f86c00096a0 tx=0x7f86c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:21.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.288+0000 7f86cf7fe700 1 -- 172.21.15.135:0/864257148 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f86d0011790 con 0x7f86d80f47e0 2023-11-15T06:40:21.494 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.491+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86d8045ad0 con 0x7f86d80f47e0 2023-11-15T06:40:21.495 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.492+0000 7f86cf7fe700 1 -- 172.21.15.135:0/864257148 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f86d0011790 con 0x7f86d80f47e0 2023-11-15T06:40:21.495 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:21.495 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:21.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f86c402ffb0 msgr2=0x7f86c4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:21.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f86c402ffb0 0x7f86c4032470 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f86c00096a0 tx=0x7f86c0006b40).stop 2023-11-15T06:40:21.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 msgr2=0x7f86d80f73d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:21.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f73d0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f86d0007bd0 tx=0x7f86d0004d10).stop 2023-11-15T06:40:21.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 shutdown_connections 2023-11-15T06:40:21.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f86c402ffb0 0x7f86c4032470 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:21.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 --2- 172.21.15.135:0/864257148 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f86d80f47e0 0x7f86d80f73d0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:21.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 >> 172.21.15.135:0/864257148 conn(0x7f86d80f0260 msgr2=0x7f86d80f0d30 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:21.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 shutdown_connections 2023-11-15T06:40:21.499 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:21.494+0000 7f86dd250700 1 -- 172.21.15.135:0/864257148 wait complete. 2023-11-15T06:40:21.499 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:22.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:20.254671+0000 mgr.smithi123.razjtn (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:22.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:21.493260+0000 mon.smithi123 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.135:0/864257148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:22.878 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:22.880 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:24.028 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 -- 172.21.15.135:0/2768492344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 msgr2=0x7f3bec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:24.029 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 --2- 172.21.15.135:0/2768492344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec0f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f3bdc009fb0 tx=0x7f3bdc009b30).stop 2023-11-15T06:40:24.029 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 -- 172.21.15.135:0/2768492344 shutdown_connections 2023-11-15T06:40:24.029 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 --2- 172.21.15.135:0/2768492344 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec0f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:24.030 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 -- 172.21.15.135:0/2768492344 >> 172.21.15.135:0/2768492344 conn(0x7f3bec0f0b70 msgr2=0x7f3bec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:24.030 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 -- 172.21.15.135:0/2768492344 shutdown_connections 2023-11-15T06:40:24.030 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 -- 172.21.15.135:0/2768492344 wait complete. 2023-11-15T06:40:24.030 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 Processor -- start 2023-11-15T06:40:24.031 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 -- start start 2023-11-15T06:40:24.031 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec1066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:24.031 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.025+0000 7f3bf37e9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3bec106ba0 con 0x7f3bec0f5410 2023-11-15T06:40:24.031 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3bf1585700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec1066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:24.031 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3bf1585700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec1066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57454/0 (socket says 172.21.15.135:57454) 2023-11-15T06:40:24.032 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3bf1585700 1 -- 172.21.15.135:0/533698933 learned_addr learned my addr 172.21.15.135:0/533698933 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:24.032 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3bf1585700 1 -- 172.21.15.135:0/533698933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bdc005040 con 0x7f3bec0f5410 2023-11-15T06:40:24.032 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3bf1585700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec1066b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3bdc013510 tx=0x7f3bdc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:24.033 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3be1ffb700 1 -- 172.21.15.135:0/533698933 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bdc009320 con 0x7f3bec0f5410 2023-11-15T06:40:24.034 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3bec106da0 con 0x7f3bec0f5410 2023-11-15T06:40:24.034 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3be1ffb700 1 -- 172.21.15.135:0/533698933 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bdc013f00 con 0x7f3bec0f5410 2023-11-15T06:40:24.034 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.026+0000 7f3be1ffb700 1 -- 172.21.15.135:0/533698933 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bdc014d90 con 0x7f3bec0f5410 2023-11-15T06:40:24.034 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.027+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3bec107230 con 0x7f3bec0f5410 2023-11-15T06:40:24.034 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.027+0000 7f3be1ffb700 1 -- 172.21.15.135:0/533698933 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bdc0263a0 con 0x7f3bec0f5410 2023-11-15T06:40:24.035 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.027+0000 7f3be1ffb700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bd802ff60 0x7f3bd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:24.035 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.028+0000 7f3be1ffb700 1 -- 172.21.15.135:0/533698933 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bdc041090 con 0x7f3bec0f5410 2023-11-15T06:40:24.035 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.028+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3bec045ad0 con 0x7f3bec0f5410 2023-11-15T06:40:24.035 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.028+0000 7f3bf0d84700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bd802ff60 0x7f3bd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:24.035 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.028+0000 7f3bf0d84700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bd802ff60 0x7f3bd8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3be80096a0 tx=0x7f3be8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:24.036 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.033+0000 7f3be1ffb700 1 -- 172.21.15.135:0/533698933 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bdc012920 con 0x7f3bec0f5410 2023-11-15T06:40:24.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.239+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3bec107f70 con 0x7f3bec0f5410 2023-11-15T06:40:24.242 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.240+0000 7f3be1ffb700 1 -- 172.21.15.135:0/533698933 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3bdc02b020 con 0x7f3bec0f5410 2023-11-15T06:40:24.243 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:24.243 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:24.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bd802ff60 msgr2=0x7f3bd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:24.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bd802ff60 0x7f3bd8032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f3be80096a0 tx=0x7f3be8006b40).stop 2023-11-15T06:40:24.244 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 msgr2=0x7f3bec1066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:24.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec1066b0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f3bdc013510 tx=0x7f3bdc00b040).stop 2023-11-15T06:40:24.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 shutdown_connections 2023-11-15T06:40:24.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bd802ff60 0x7f3bd8032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:24.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 --2- 172.21.15.135:0/533698933 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3bec0f5410 0x7f3bec1066b0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:24.245 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 >> 172.21.15.135:0/533698933 conn(0x7f3bec0f0b70 msgr2=0x7f3bec0f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:24.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 shutdown_connections 2023-11-15T06:40:24.246 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:24.241+0000 7f3bf37e9700 1 -- 172.21.15.135:0/533698933 wait complete. 2023-11-15T06:40:24.246 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:24.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:22.254958+0000 mgr.smithi123.razjtn (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:25.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:24 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:24.240971+0000 mon.smithi123 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.135:0/533698933' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:25.624 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:25.624 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:26.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:24.255295+0000 mgr.smithi123.razjtn (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:26.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.761+0000 7f4cb076b700 1 -- 172.21.15.135:0/1357135840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 msgr2=0x7f4ca80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:26.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.761+0000 7f4cb076b700 1 --2- 172.21.15.135:0/1357135840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca80f5830 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f4c98009fb0 tx=0x7f4c98009b30).stop 2023-11-15T06:40:26.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.761+0000 7f4cb076b700 1 -- 172.21.15.135:0/1357135840 shutdown_connections 2023-11-15T06:40:26.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.761+0000 7f4cb076b700 1 --2- 172.21.15.135:0/1357135840 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca80f5830 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:26.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.761+0000 7f4cb076b700 1 -- 172.21.15.135:0/1357135840 >> 172.21.15.135:0/1357135840 conn(0x7f4ca80f0b70 msgr2=0x7f4ca80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:26.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.761+0000 7f4cb076b700 1 -- 172.21.15.135:0/1357135840 shutdown_connections 2023-11-15T06:40:26.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.761+0000 7f4cb076b700 1 -- 172.21.15.135:0/1357135840 wait complete. 2023-11-15T06:40:26.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cb076b700 1 Processor -- start 2023-11-15T06:40:26.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cb076b700 1 -- start start 2023-11-15T06:40:26.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cb076b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca81093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:26.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cb076b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca8060280 con 0x7f4ca80f5410 2023-11-15T06:40:26.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cae507700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca81093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:26.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cae507700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca81093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57472/0 (socket says 172.21.15.135:57472) 2023-11-15T06:40:26.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cae507700 1 -- 172.21.15.135:0/1160657025 learned_addr learned my addr 172.21.15.135:0/1160657025 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:26.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cae507700 1 -- 172.21.15.135:0/1160657025 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c98005040 con 0x7f4ca80f5410 2023-11-15T06:40:26.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4cae507700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca81093f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4c98013510 tx=0x7f4c9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:26.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.762+0000 7f4c9effd700 1 -- 172.21.15.135:0/1160657025 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c98009320 con 0x7f4ca80f5410 2023-11-15T06:40:26.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4c9effd700 1 -- 172.21.15.135:0/1160657025 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c98013f00 con 0x7f4ca80f5410 2023-11-15T06:40:26.769 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca8060480 con 0x7f4ca80f5410 2023-11-15T06:40:26.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4c9effd700 1 -- 172.21.15.135:0/1160657025 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c98014d90 con 0x7f4ca80f5410 2023-11-15T06:40:26.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca8109b20 con 0x7f4ca80f5410 2023-11-15T06:40:26.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4c9effd700 1 -- 172.21.15.135:0/1160657025 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c98014420 con 0x7f4ca80f5410 2023-11-15T06:40:26.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca8045ad0 con 0x7f4ca80f5410 2023-11-15T06:40:26.770 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4c9effd700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c9402ff60 0x7f4c94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:26.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.763+0000 7f4c9effd700 1 -- 172.21.15.135:0/1160657025 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c98041080 con 0x7f4ca80f5410 2023-11-15T06:40:26.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.764+0000 7f4cadd06700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c9402ff60 0x7f4c94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:26.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.764+0000 7f4cadd06700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c9402ff60 0x7f4c94032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4ca40096a0 tx=0x7f4ca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:26.771 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.768+0000 7f4c9effd700 1 -- 172.21.15.135:0/1160657025 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c980095f0 con 0x7f4ca80f5410 2023-11-15T06:40:26.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.977+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ca810a340 con 0x7f4ca80f5410 2023-11-15T06:40:26.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.978+0000 7f4c9effd700 1 -- 172.21.15.135:0/1160657025 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4c9801db70 con 0x7f4ca80f5410 2023-11-15T06:40:26.981 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:26.981 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:26.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c9402ff60 msgr2=0x7f4c94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:26.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c9402ff60 0x7f4c94032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f4ca40096a0 tx=0x7f4ca4006b40).stop 2023-11-15T06:40:26.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 msgr2=0x7f4ca81093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:26.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca81093f0 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f4c98013510 tx=0x7f4c9800b040).stop 2023-11-15T06:40:26.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 shutdown_connections 2023-11-15T06:40:26.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c9402ff60 0x7f4c94032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:26.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 --2- 172.21.15.135:0/1160657025 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4ca80f5410 0x7f4ca81093f0 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:26.984 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 >> 172.21.15.135:0/1160657025 conn(0x7f4ca80f0b70 msgr2=0x7f4ca80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:26.984 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 shutdown_connections 2023-11-15T06:40:26.984 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:26.979+0000 7f4cb076b700 1 -- 172.21.15.135:0/1160657025 wait complete. 2023-11-15T06:40:26.985 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:28.354 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:28.355 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:28.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:26.255677+0000 mgr.smithi123.razjtn (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:28.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:26.978881+0000 mon.smithi123 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.135:0/1160657025' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:29.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 -- 172.21.15.135:0/485312613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 msgr2=0x7f32300f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:29.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 --2- 172.21.15.135:0/485312613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32300f55c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f3224009fb0 tx=0x7f3224009b30).stop 2023-11-15T06:40:29.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 -- 172.21.15.135:0/485312613 shutdown_connections 2023-11-15T06:40:29.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 --2- 172.21.15.135:0/485312613 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32300f55c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:29.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 -- 172.21.15.135:0/485312613 >> 172.21.15.135:0/485312613 conn(0x7f32300f09a0 msgr2=0x7f32300f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:29.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 -- 172.21.15.135:0/485312613 shutdown_connections 2023-11-15T06:40:29.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 -- 172.21.15.135:0/485312613 wait complete. 2023-11-15T06:40:29.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 Processor -- start 2023-11-15T06:40:29.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.482+0000 7f32359a0700 1 -- start start 2023-11-15T06:40:29.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.483+0000 7f32359a0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32301091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:29.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.483+0000 7f32359a0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3230060240 con 0x7f32300f51a0 2023-11-15T06:40:29.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.483+0000 7f323499e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32301091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:29.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.483+0000 7f323499e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32301091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54198/0 (socket says 172.21.15.135:54198) 2023-11-15T06:40:29.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.483+0000 7f323499e700 1 -- 172.21.15.135:0/1096785565 learned_addr learned my addr 172.21.15.135:0/1096785565 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:29.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.483+0000 7f323499e700 1 -- 172.21.15.135:0/1096785565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3224005040 con 0x7f32300f51a0 2023-11-15T06:40:29.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.483+0000 7f323499e700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32301091b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3224013510 tx=0x7f322400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:29.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.484+0000 7f322d7fa700 1 -- 172.21.15.135:0/1096785565 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224009320 con 0x7f32300f51a0 2023-11-15T06:40:29.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.484+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3230060440 con 0x7f32300f51a0 2023-11-15T06:40:29.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.484+0000 7f322d7fa700 1 -- 172.21.15.135:0/1096785565 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3224013f00 con 0x7f32300f51a0 2023-11-15T06:40:29.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.484+0000 7f322d7fa700 1 -- 172.21.15.135:0/1096785565 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3224014dc0 con 0x7f32300f51a0 2023-11-15T06:40:29.490 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.484+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3230109860 con 0x7f32300f51a0 2023-11-15T06:40:29.491 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.484+0000 7f322d7fa700 1 -- 172.21.15.135:0/1096785565 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32240095f0 con 0x7f32300f51a0 2023-11-15T06:40:29.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.484+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3230045ad0 con 0x7f32300f51a0 2023-11-15T06:40:29.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.485+0000 7f322d7fa700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f322002ff60 0x7f3220032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:29.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.485+0000 7f322d7fa700 1 -- 172.21.15.135:0/1096785565 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32240410c0 con 0x7f32300f51a0 2023-11-15T06:40:29.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.485+0000 7f322ffff700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f322002ff60 0x7f3220032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:29.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.485+0000 7f322ffff700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f322002ff60 0x7f3220032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f321c0096a0 tx=0x7f321c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:29.493 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.489+0000 7f322d7fa700 1 -- 172.21.15.135:0/1096785565 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f322401c790 con 0x7f32300f51a0 2023-11-15T06:40:29.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.696+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f323010a080 con 0x7f32300f51a0 2023-11-15T06:40:29.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.697+0000 7f322d7fa700 1 -- 172.21.15.135:0/1096785565 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3224010230 con 0x7f32300f51a0 2023-11-15T06:40:29.700 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:29.700 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:29.701 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f322002ff60 msgr2=0x7f3220032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:29.701 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f322002ff60 0x7f3220032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f321c0096a0 tx=0x7f321c006b40).stop 2023-11-15T06:40:29.702 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 msgr2=0x7f32301091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:29.702 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32301091b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f3224013510 tx=0x7f322400b040).stop 2023-11-15T06:40:29.702 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 shutdown_connections 2023-11-15T06:40:29.702 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f322002ff60 0x7f3220032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:29.703 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 --2- 172.21.15.135:0/1096785565 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f32300f51a0 0x7f32301091b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:29.703 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 >> 172.21.15.135:0/1096785565 conn(0x7f32300f09a0 msgr2=0x7f32300f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:29.703 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 shutdown_connections 2023-11-15T06:40:29.703 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:29.699+0000 7f32359a0700 1 -- 172.21.15.135:0/1096785565 wait complete. 2023-11-15T06:40:29.704 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:30.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:28.255988+0000 mgr.smithi123.razjtn (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:30.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:29.698107+0000 mon.smithi123 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.135:0/1096785565' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:31.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:31.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:32.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 -- 172.21.15.135:0/3010337128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 msgr2=0x7f8f580f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:32.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 --2- 172.21.15.135:0/3010337128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f580f90f0 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f8f40009fb0 tx=0x7f8f40009b30).stop 2023-11-15T06:40:32.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 -- 172.21.15.135:0/3010337128 shutdown_connections 2023-11-15T06:40:32.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 --2- 172.21.15.135:0/3010337128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f580f90f0 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:32.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 -- 172.21.15.135:0/3010337128 >> 172.21.15.135:0/3010337128 conn(0x7f8f580f0b70 msgr2=0x7f8f580f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:32.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 -- 172.21.15.135:0/3010337128 shutdown_connections 2023-11-15T06:40:32.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 -- 172.21.15.135:0/3010337128 wait complete. 2023-11-15T06:40:32.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.224+0000 7f8f5dbed700 1 Processor -- start 2023-11-15T06:40:32.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f5dbed700 1 -- start start 2023-11-15T06:40:32.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f5dbed700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f581066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:32.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f5dbed700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f58106bd0 con 0x7f8f58060280 2023-11-15T06:40:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f577fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f581066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f577fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f581066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54206/0 (socket says 172.21.15.135:54206) 2023-11-15T06:40:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f577fe700 1 -- 172.21.15.135:0/539651944 learned_addr learned my addr 172.21.15.135:0/539651944 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f577fe700 1 -- 172.21.15.135:0/539651944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f40005040 con 0x7f8f58060280 2023-11-15T06:40:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.225+0000 7f8f577fe700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f581066e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8f40013510 tx=0x7f8f4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.226+0000 7f8f4ffff700 1 -- 172.21.15.135:0/539651944 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f40009320 con 0x7f8f58060280 2023-11-15T06:40:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.226+0000 7f8f4ffff700 1 -- 172.21.15.135:0/539651944 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f40013f00 con 0x7f8f58060280 2023-11-15T06:40:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.226+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f58106dd0 con 0x7f8f58060280 2023-11-15T06:40:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.226+0000 7f8f4ffff700 1 -- 172.21.15.135:0/539651944 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f40014dc0 con 0x7f8f58060280 2023-11-15T06:40:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.226+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f58107260 con 0x7f8f58060280 2023-11-15T06:40:32.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.226+0000 7f8f4ffff700 1 -- 172.21.15.135:0/539651944 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f4001d3d0 con 0x7f8f58060280 2023-11-15T06:40:32.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.227+0000 7f8f4ffff700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f4402ff60 0x7f8f44032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:32.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.227+0000 7f8f4ffff700 1 -- 172.21.15.135:0/539651944 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f400410d0 con 0x7f8f58060280 2023-11-15T06:40:32.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.227+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f58045ad0 con 0x7f8f58060280 2023-11-15T06:40:32.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.227+0000 7f8f56ffd700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f4402ff60 0x7f8f44032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:32.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.227+0000 7f8f56ffd700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f4402ff60 0x7f8f44032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8f480096a0 tx=0x7f8f48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:32.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.232+0000 7f8f4ffff700 1 -- 172.21.15.135:0/539651944 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f40012900 con 0x7f8f58060280 2023-11-15T06:40:32.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:30.256379+0000 mgr.smithi123.razjtn (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:32.445 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.442+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f58107ee0 con 0x7f8f58060280 2023-11-15T06:40:32.445 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.443+0000 7f8f4ffff700 1 -- 172.21.15.135:0/539651944 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f4002b020 con 0x7f8f58060280 2023-11-15T06:40:32.446 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:32.446 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:32.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f4402ff60 msgr2=0x7f8f44032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:32.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f4402ff60 0x7f8f44032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f8f480096a0 tx=0x7f8f48006b40).stop 2023-11-15T06:40:32.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 msgr2=0x7f8f581066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:32.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f581066e0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f8f40013510 tx=0x7f8f4000b040).stop 2023-11-15T06:40:32.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 shutdown_connections 2023-11-15T06:40:32.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f4402ff60 0x7f8f44032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:32.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 --2- 172.21.15.135:0/539651944 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f58060280 0x7f8f581066e0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:32.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 >> 172.21.15.135:0/539651944 conn(0x7f8f580f0b70 msgr2=0x7f8f580f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:32.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 shutdown_connections 2023-11-15T06:40:32.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:32.444+0000 7f8f5dbed700 1 -- 172.21.15.135:0/539651944 wait complete. 2023-11-15T06:40:32.449 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:33.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:32.443825+0000 mon.smithi123 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.135:0/539651944' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:33.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:33.832 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:34.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:32.256690+0000 mgr.smithi123.razjtn (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:34.926 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 -- 172.21.15.135:0/431691207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 msgr2=0x7f5b180f8fe0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:34.926 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 --2- 172.21.15.135:0/431691207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b180f8fe0 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7f5b0c009fb0 tx=0x7f5b0c009b30).stop 2023-11-15T06:40:34.926 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 -- 172.21.15.135:0/431691207 shutdown_connections 2023-11-15T06:40:34.927 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 --2- 172.21.15.135:0/431691207 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b180f8fe0 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:34.927 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 -- 172.21.15.135:0/431691207 >> 172.21.15.135:0/431691207 conn(0x7f5b180f09a0 msgr2=0x7f5b180f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:34.927 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 -- 172.21.15.135:0/431691207 shutdown_connections 2023-11-15T06:40:34.927 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 -- 172.21.15.135:0/431691207 wait complete. 2023-11-15T06:40:34.927 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 Processor -- start 2023-11-15T06:40:34.928 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.923+0000 7f5b1de4c700 1 -- start start 2023-11-15T06:40:34.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.924+0000 7f5b1de4c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b181064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:34.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.924+0000 7f5b1de4c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5b181069c0 con 0x7f5b18060120 2023-11-15T06:40:34.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.924+0000 7f5b1ce4a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b181064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:34.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.924+0000 7f5b1ce4a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b181064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54228/0 (socket says 172.21.15.135:54228) 2023-11-15T06:40:34.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.924+0000 7f5b1ce4a700 1 -- 172.21.15.135:0/2239368338 learned_addr learned my addr 172.21.15.135:0/2239368338 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:34.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.924+0000 7f5b1ce4a700 1 -- 172.21.15.135:0/2239368338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5b0c005040 con 0x7f5b18060120 2023-11-15T06:40:34.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.924+0000 7f5b1ce4a700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b181064d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5b0c013510 tx=0x7f5b0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:34.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.925+0000 7f5b157fa700 1 -- 172.21.15.135:0/2239368338 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0c009320 con 0x7f5b18060120 2023-11-15T06:40:34.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.925+0000 7f5b157fa700 1 -- 172.21.15.135:0/2239368338 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5b0c013f00 con 0x7f5b18060120 2023-11-15T06:40:34.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.925+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5b18106bc0 con 0x7f5b18060120 2023-11-15T06:40:34.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.925+0000 7f5b157fa700 1 -- 172.21.15.135:0/2239368338 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5b0c014d90 con 0x7f5b18060120 2023-11-15T06:40:34.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.925+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5b18107050 con 0x7f5b18060120 2023-11-15T06:40:34.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.925+0000 7f5b157fa700 1 -- 172.21.15.135:0/2239368338 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5b0c014420 con 0x7f5b18060120 2023-11-15T06:40:34.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.926+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5b18045ad0 con 0x7f5b18060120 2023-11-15T06:40:34.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.926+0000 7f5b157fa700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b0802ff60 0x7f5b08032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:34.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.926+0000 7f5b157fa700 1 -- 172.21.15.135:0/2239368338 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5b0c040ff0 con 0x7f5b18060120 2023-11-15T06:40:34.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.926+0000 7f5b17fff700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b0802ff60 0x7f5b08032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:34.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.926+0000 7f5b17fff700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b0802ff60 0x7f5b08032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5b040096a0 tx=0x7f5b04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:34.934 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:34.930+0000 7f5b157fa700 1 -- 172.21.15.135:0/2239368338 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5b0c0095f0 con 0x7f5b18060120 2023-11-15T06:40:35.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.135+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5b18107f00 con 0x7f5b18060120 2023-11-15T06:40:35.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.135+0000 7f5b157fa700 1 -- 172.21.15.135:0/2239368338 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5b0c01db70 con 0x7f5b18060120 2023-11-15T06:40:35.139 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:35.139 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:35.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b0802ff60 msgr2=0x7f5b08032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:35.140 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b0802ff60 0x7f5b08032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f5b040096a0 tx=0x7f5b04006b40).stop 2023-11-15T06:40:35.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 msgr2=0x7f5b181064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:35.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b181064d0 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f5b0c013510 tx=0x7f5b0c00b040).stop 2023-11-15T06:40:35.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 shutdown_connections 2023-11-15T06:40:35.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5b0802ff60 0x7f5b08032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:35.141 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 --2- 172.21.15.135:0/2239368338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5b18060120 0x7f5b181064d0 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:35.142 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.137+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 >> 172.21.15.135:0/2239368338 conn(0x7f5b180f09a0 msgr2=0x7f5b180f1450 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:35.142 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.138+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 shutdown_connections 2023-11-15T06:40:35.142 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:35.138+0000 7f5b1de4c700 1 -- 172.21.15.135:0/2239368338 wait complete. 2023-11-15T06:40:35.143 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:36.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:34.256913+0000 mgr.smithi123.razjtn (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:36.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:35.136539+0000 mon.smithi123 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.135:0/2239368338' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:36.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:36.522 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:37.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.659+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1030862435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 msgr2=0x7fdfe40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:37.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.659+0000 7fdfeaec6700 1 --2- 172.21.15.135:0/1030862435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe40f66c0 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7fdfd4009fb0 tx=0x7fdfd4009b30).stop 2023-11-15T06:40:37.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.659+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1030862435 shutdown_connections 2023-11-15T06:40:37.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.659+0000 7fdfeaec6700 1 --2- 172.21.15.135:0/1030862435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe40f66c0 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:37.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.659+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1030862435 >> 172.21.15.135:0/1030862435 conn(0x7fdfe40f1aa0 msgr2=0x7fdfe40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:37.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.660+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1030862435 shutdown_connections 2023-11-15T06:40:37.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.660+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1030862435 wait complete. 2023-11-15T06:40:37.665 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.660+0000 7fdfeaec6700 1 Processor -- start 2023-11-15T06:40:37.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.660+0000 7fdfeaec6700 1 -- start start 2023-11-15T06:40:37.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfeaec6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:37.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfeaec6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdfe41069a0 con 0x7fdfe40f62a0 2023-11-15T06:40:37.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfe9ec4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:37.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfe9ec4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:54248/0 (socket says 172.21.15.135:54248) 2023-11-15T06:40:37.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfe9ec4700 1 -- 172.21.15.135:0/1248388470 learned_addr learned my addr 172.21.15.135:0/1248388470 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:37.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfe9ec4700 1 -- 172.21.15.135:0/1248388470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdfd4005040 con 0x7fdfe40f62a0 2023-11-15T06:40:37.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfe9ec4700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe41064b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fdfd4013510 tx=0x7fdfd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:37.669 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfda7fc700 1 -- 172.21.15.135:0/1248388470 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd4009320 con 0x7fdfe40f62a0 2023-11-15T06:40:37.669 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfda7fc700 1 -- 172.21.15.135:0/1248388470 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdfd4013f00 con 0x7fdfe40f62a0 2023-11-15T06:40:37.669 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdfe4106ba0 con 0x7fdfe40f62a0 2023-11-15T06:40:37.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfda7fc700 1 -- 172.21.15.135:0/1248388470 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdfd4014dc0 con 0x7fdfe40f62a0 2023-11-15T06:40:37.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.661+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdfe4107030 con 0x7fdfe40f62a0 2023-11-15T06:40:37.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.662+0000 7fdfda7fc700 1 -- 172.21.15.135:0/1248388470 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdfd401d3d0 con 0x7fdfe40f62a0 2023-11-15T06:40:37.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.662+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdfe4045ad0 con 0x7fdfe40f62a0 2023-11-15T06:40:37.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.662+0000 7fdfda7fc700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdfd002ff60 0x7fdfd0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:37.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.662+0000 7fdfda7fc700 1 -- 172.21.15.135:0/1248388470 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdfd40413b0 con 0x7fdfe40f62a0 2023-11-15T06:40:37.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.663+0000 7fdfe96c3700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdfd002ff60 0x7fdfd0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:37.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.663+0000 7fdfe96c3700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdfd002ff60 0x7fdfd0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:37.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.667+0000 7fdfda7fc700 1 -- 172.21.15.135:0/1248388470 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdfd4012510 con 0x7fdfe40f62a0 2023-11-15T06:40:37.881 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.878+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdfe4107ee0 con 0x7fdfe40f62a0 2023-11-15T06:40:37.881 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.879+0000 7fdfda7fc700 1 -- 172.21.15.135:0/1248388470 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdfd4012330 con 0x7fdfe40f62a0 2023-11-15T06:40:37.882 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:37.882 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:37.883 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdfd002ff60 msgr2=0x7fdfd0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:37.883 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdfd002ff60 0x7fdfd0032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7fdfe00096a0 tx=0x7fdfe0006b40).stop 2023-11-15T06:40:37.883 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 msgr2=0x7fdfe41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:37.884 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe41064b0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7fdfd4013510 tx=0x7fdfd400b040).stop 2023-11-15T06:40:37.884 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 shutdown_connections 2023-11-15T06:40:37.884 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdfd002ff60 0x7fdfd0032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:37.884 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 --2- 172.21.15.135:0/1248388470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdfe40f62a0 0x7fdfe41064b0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:37.884 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 >> 172.21.15.135:0/1248388470 conn(0x7fdfe40f1aa0 msgr2=0x7fdfe40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:37.885 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 shutdown_connections 2023-11-15T06:40:37.885 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:37.880+0000 7fdfeaec6700 1 -- 172.21.15.135:0/1248388470 wait complete. 2023-11-15T06:40:37.885 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:38.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:36.257185+0000 mgr.smithi123.razjtn (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:38.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:37.879899+0000 mon.smithi123 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.135:0/1248388470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:39.260 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:39.261 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:40.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:38.257511+0000 mgr.smithi123.razjtn (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:40.411 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 -- 172.21.15.135:0/70787377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 msgr2=0x7ff1300f66e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:40.412 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 --2- 172.21.15.135:0/70787377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff1300f66e0 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff120009fb0 tx=0x7ff120009b30).stop 2023-11-15T06:40:40.412 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 -- 172.21.15.135:0/70787377 shutdown_connections 2023-11-15T06:40:40.412 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 --2- 172.21.15.135:0/70787377 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff1300f66e0 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:40.412 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 -- 172.21.15.135:0/70787377 >> 172.21.15.135:0/70787377 conn(0x7ff1300f1ac0 msgr2=0x7ff1300f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:40.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 -- 172.21.15.135:0/70787377 shutdown_connections 2023-11-15T06:40:40.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 -- 172.21.15.135:0/70787377 wait complete. 2023-11-15T06:40:40.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 Processor -- start 2023-11-15T06:40:40.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 -- start start 2023-11-15T06:40:40.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.408+0000 7ff1349fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff130106520 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:40.414 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff1349fe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff130106a10 con 0x7ff1300f62c0 2023-11-15T06:40:40.414 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff12f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff130106520 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:40.414 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff12f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff130106520 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37044/0 (socket says 172.21.15.135:37044) 2023-11-15T06:40:40.414 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff12f7fe700 1 -- 172.21.15.135:0/58840353 learned_addr learned my addr 172.21.15.135:0/58840353 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:40.414 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff12f7fe700 1 -- 172.21.15.135:0/58840353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff120005040 con 0x7ff1300f62c0 2023-11-15T06:40:40.415 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff12f7fe700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff130106520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff120013510 tx=0x7ff12000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:40.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff117fff700 1 -- 172.21.15.135:0/58840353 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff120009320 con 0x7ff1300f62c0 2023-11-15T06:40:40.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff130106c10 con 0x7ff1300f62c0 2023-11-15T06:40:40.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff117fff700 1 -- 172.21.15.135:0/58840353 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff120013f00 con 0x7ff1300f62c0 2023-11-15T06:40:40.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff117fff700 1 -- 172.21.15.135:0/58840353 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff120014dc0 con 0x7ff1300f62c0 2023-11-15T06:40:40.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.409+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff1301070a0 con 0x7ff1300f62c0 2023-11-15T06:40:40.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.410+0000 7ff117fff700 1 -- 172.21.15.135:0/58840353 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff120014420 con 0x7ff1300f62c0 2023-11-15T06:40:40.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.410+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff130045ad0 con 0x7ff1300f62c0 2023-11-15T06:40:40.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.410+0000 7ff117fff700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ff11802ff60 0x7ff118032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:40.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.410+0000 7ff117fff700 1 -- 172.21.15.135:0/58840353 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff120041050 con 0x7ff1300f62c0 2023-11-15T06:40:40.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.410+0000 7ff12effd700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ff11802ff60 0x7ff118032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:40.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.411+0000 7ff12effd700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ff11802ff60 0x7ff118032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff1240096a0 tx=0x7ff124006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:40.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.415+0000 7ff117fff700 1 -- 172.21.15.135:0/58840353 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff1200095f0 con 0x7ff1300f62c0 2023-11-15T06:40:40.625 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.623+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff130107f50 con 0x7ff1300f62c0 2023-11-15T06:40:40.626 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.623+0000 7ff117fff700 1 -- 172.21.15.135:0/58840353 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff12001cb10 con 0x7ff1300f62c0 2023-11-15T06:40:40.627 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:40.627 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:40.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ff11802ff60 msgr2=0x7ff118032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:40.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ff11802ff60 0x7ff118032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff1240096a0 tx=0x7ff124006b40).stop 2023-11-15T06:40:40.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 msgr2=0x7ff130106520 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:40.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff130106520 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff120013510 tx=0x7ff12000b040).stop 2023-11-15T06:40:40.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 shutdown_connections 2023-11-15T06:40:40.629 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ff11802ff60 0x7ff118032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:40.629 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 --2- 172.21.15.135:0/58840353 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ff1300f62c0 0x7ff130106520 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:40.629 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 >> 172.21.15.135:0/58840353 conn(0x7ff1300f1ac0 msgr2=0x7ff1300f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:40.629 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 shutdown_connections 2023-11-15T06:40:40.629 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:40.625+0000 7ff1349fe700 1 -- 172.21.15.135:0/58840353 wait complete. 2023-11-15T06:40:40.630 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:41.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:40.364512+0000 mon.smithi123 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:40:41.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:40.371347+0000 mon.smithi123 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:40:41.407 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:40.624652+0000 mon.smithi123 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.135:0/58840353' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:42.007 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:42.007 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:42.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:40.257862+0000 mgr.smithi123.razjtn (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:43.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 -- 172.21.15.135:0/3992859307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 msgr2=0x7f6cb40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:43.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 --2- 172.21.15.135:0/3992859307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb40f66c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7f6ca4009fb0 tx=0x7f6ca4009b30).stop 2023-11-15T06:40:43.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 -- 172.21.15.135:0/3992859307 shutdown_connections 2023-11-15T06:40:43.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 --2- 172.21.15.135:0/3992859307 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb40f66c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:43.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 -- 172.21.15.135:0/3992859307 >> 172.21.15.135:0/3992859307 conn(0x7f6cb40f1aa0 msgr2=0x7f6cb40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:43.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 -- 172.21.15.135:0/3992859307 shutdown_connections 2023-11-15T06:40:43.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 -- 172.21.15.135:0/3992859307 wait complete. 2023-11-15T06:40:43.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 Processor -- start 2023-11-15T06:40:43.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.199+0000 7f6cb359e700 1 -- start start 2023-11-15T06:40:43.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.200+0000 7f6cb359e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:43.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.200+0000 7f6cb359e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6cb41069a0 con 0x7f6cb40f62a0 2023-11-15T06:40:43.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.200+0000 7f6cb259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:43.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.200+0000 7f6cb259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37072/0 (socket says 172.21.15.135:37072) 2023-11-15T06:40:43.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.200+0000 7f6cb259c700 1 -- 172.21.15.135:0/2314790294 learned_addr learned my addr 172.21.15.135:0/2314790294 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.200+0000 7f6cb259c700 1 -- 172.21.15.135:0/2314790294 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6ca4005040 con 0x7f6cb40f62a0 2023-11-15T06:40:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.200+0000 7f6cb259c700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb41064b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f6ca4013510 tx=0x7f6ca400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.201+0000 7f6ca2ffd700 1 -- 172.21.15.135:0/2314790294 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca4009320 con 0x7f6cb40f62a0 2023-11-15T06:40:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.201+0000 7f6ca2ffd700 1 -- 172.21.15.135:0/2314790294 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6ca4013f00 con 0x7f6cb40f62a0 2023-11-15T06:40:43.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.201+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6cb4106ba0 con 0x7f6cb40f62a0 2023-11-15T06:40:43.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.201+0000 7f6ca2ffd700 1 -- 172.21.15.135:0/2314790294 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6ca4014dc0 con 0x7f6cb40f62a0 2023-11-15T06:40:43.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.201+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6cb4107030 con 0x7f6cb40f62a0 2023-11-15T06:40:43.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.201+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6cb4045ad0 con 0x7f6cb40f62a0 2023-11-15T06:40:43.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.202+0000 7f6ca2ffd700 1 -- 172.21.15.135:0/2314790294 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6ca40095f0 con 0x7f6cb40f62a0 2023-11-15T06:40:43.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.202+0000 7f6ca2ffd700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c9c02ff60 0x7f6c9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:43.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.202+0000 7f6ca2ffd700 1 -- 172.21.15.135:0/2314790294 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6ca4041080 con 0x7f6cb40f62a0 2023-11-15T06:40:43.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.202+0000 7f6cb1d9b700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c9c02ff60 0x7f6c9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:43.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.202+0000 7f6cb1d9b700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c9c02ff60 0x7f6c9c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f6ca80096a0 tx=0x7f6ca8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:43.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.207+0000 7f6ca2ffd700 1 -- 172.21.15.135:0/2314790294 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6ca401c790 con 0x7f6cb40f62a0 2023-11-15T06:40:43.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.414+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6cb4107ee0 con 0x7f6cb40f62a0 2023-11-15T06:40:43.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.415+0000 7f6ca2ffd700 1 -- 172.21.15.135:0/2314790294 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6ca4010230 con 0x7f6cb40f62a0 2023-11-15T06:40:43.418 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:43.418 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:43.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c9c02ff60 msgr2=0x7f6c9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:43.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c9c02ff60 0x7f6c9c032420 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f6ca80096a0 tx=0x7f6ca8006b40).stop 2023-11-15T06:40:43.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 msgr2=0x7f6cb41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:43.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb41064b0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7f6ca4013510 tx=0x7f6ca400b040).stop 2023-11-15T06:40:43.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 shutdown_connections 2023-11-15T06:40:43.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c9c02ff60 0x7f6c9c032420 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:43.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 --2- 172.21.15.135:0/2314790294 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6cb40f62a0 0x7f6cb41064b0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:43.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 >> 172.21.15.135:0/2314790294 conn(0x7f6cb40f1aa0 msgr2=0x7f6cb40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:43.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 shutdown_connections 2023-11-15T06:40:43.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:43.416+0000 7f6cb359e700 1 -- 172.21.15.135:0/2314790294 wait complete. 2023-11-15T06:40:43.421 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:44.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:42.258220+0000 mgr.smithi123.razjtn (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:44.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:43.415784+0000 mon.smithi123 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.135:0/2314790294' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:44.801 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:44.802 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:45.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 -- 172.21.15.135:0/3595719172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 msgr2=0x7ffb5c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:45.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 --2- 172.21.15.135:0/3595719172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c0f66c0 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7ffb50009fb0 tx=0x7ffb50009b30).stop 2023-11-15T06:40:45.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 -- 172.21.15.135:0/3595719172 shutdown_connections 2023-11-15T06:40:45.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 --2- 172.21.15.135:0/3595719172 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c0f66c0 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:45.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 -- 172.21.15.135:0/3595719172 >> 172.21.15.135:0/3595719172 conn(0x7ffb5c0f1aa0 msgr2=0x7ffb5c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:45.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 -- 172.21.15.135:0/3595719172 shutdown_connections 2023-11-15T06:40:45.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 -- 172.21.15.135:0/3595719172 wait complete. 2023-11-15T06:40:45.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 Processor -- start 2023-11-15T06:40:45.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 -- start start 2023-11-15T06:40:45.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.914+0000 7ffb61b9e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:45.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb61b9e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb5c069e20 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb60b9c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:45.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb60b9c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37088/0 (socket says 172.21.15.135:37088) 2023-11-15T06:40:45.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb60b9c700 1 -- 172.21.15.135:0/938612536 learned_addr learned my addr 172.21.15.135:0/938612536 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:45.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb60b9c700 1 -- 172.21.15.135:0/938612536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb50005040 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb60b9c700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c06b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ffb50013510 tx=0x7ffb5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:45.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb597fa700 1 -- 172.21.15.135:0/938612536 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb50009320 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb5c069fc0 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.915+0000 7ffb597fa700 1 -- 172.21.15.135:0/938612536 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb50013f00 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.916+0000 7ffb597fa700 1 -- 172.21.15.135:0/938612536 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb50014dc0 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.916+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb5c06a450 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.916+0000 7ffb597fa700 1 -- 172.21.15.135:0/938612536 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb500095f0 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.916+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb5c059100 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.916+0000 7ffb597fa700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb4c02ff60 0x7ffb4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:45.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.916+0000 7ffb597fa700 1 -- 172.21.15.135:0/938612536 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb50041030 con 0x7ffb5c0f62a0 2023-11-15T06:40:45.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.917+0000 7ffb5bfff700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb4c02ff60 0x7ffb4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:45.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.917+0000 7ffb5bfff700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb4c02ff60 0x7ffb4c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffb480096a0 tx=0x7ffb48006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:45.924 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:45.921+0000 7ffb597fa700 1 -- 172.21.15.135:0/938612536 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb500268c0 con 0x7ffb5c0f62a0 2023-11-15T06:40:46.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.126+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb5c0fbce0 con 0x7ffb5c0f62a0 2023-11-15T06:40:46.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.127+0000 7ffb597fa700 1 -- 172.21.15.135:0/938612536 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb50010230 con 0x7ffb5c0f62a0 2023-11-15T06:40:46.130 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:46.130 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:46.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.128+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb4c02ff60 msgr2=0x7ffb4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:46.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.128+0000 7ffb61b9e700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb4c02ff60 0x7ffb4c032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7ffb480096a0 tx=0x7ffb48006b40).stop 2023-11-15T06:40:46.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.128+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 msgr2=0x7ffb5c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:46.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.128+0000 7ffb61b9e700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c06b9a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7ffb50013510 tx=0x7ffb5000b040).stop 2023-11-15T06:40:46.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.128+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 shutdown_connections 2023-11-15T06:40:46.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.128+0000 7ffb61b9e700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb4c02ff60 0x7ffb4c032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:46.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.128+0000 7ffb61b9e700 1 --2- 172.21.15.135:0/938612536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb5c0f62a0 0x7ffb5c06b9a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:46.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.129+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 >> 172.21.15.135:0/938612536 conn(0x7ffb5c0f1aa0 msgr2=0x7ffb5c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:46.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.129+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 shutdown_connections 2023-11-15T06:40:46.133 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:46.129+0000 7ffb61b9e700 1 -- 172.21.15.135:0/938612536 wait complete. 2023-11-15T06:40:46.133 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:46.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:44.258475+0000 mgr.smithi123.razjtn (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:47.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:46 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:46.127827+0000 mon.smithi123 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.135:0/938612536' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:47.497 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:47.498 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:48.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:46.258845+0000 mgr.smithi123.razjtn (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:48.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.671+0000 7fccc5d1c700 1 -- 172.21.15.135:0/3700618424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 msgr2=0x7fccc00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:48.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.671+0000 7fccc5d1c700 1 --2- 172.21.15.135:0/3700618424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc00f6930 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7fccb4009fb0 tx=0x7fccb4009b30).stop 2023-11-15T06:40:48.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.671+0000 7fccc5d1c700 1 -- 172.21.15.135:0/3700618424 shutdown_connections 2023-11-15T06:40:48.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.671+0000 7fccc5d1c700 1 --2- 172.21.15.135:0/3700618424 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc00f6930 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:48.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.671+0000 7fccc5d1c700 1 -- 172.21.15.135:0/3700618424 >> 172.21.15.135:0/3700618424 conn(0x7fccc00f1c70 msgr2=0x7fccc00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:48.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.671+0000 7fccc5d1c700 1 -- 172.21.15.135:0/3700618424 shutdown_connections 2023-11-15T06:40:48.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.671+0000 7fccc5d1c700 1 -- 172.21.15.135:0/3700618424 wait complete. 2023-11-15T06:40:48.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.672+0000 7fccc5d1c700 1 Processor -- start 2023-11-15T06:40:48.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.672+0000 7fccc5d1c700 1 -- start start 2023-11-15T06:40:48.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.672+0000 7fccc5d1c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc01066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:48.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.672+0000 7fccc5d1c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccc0106bc0 con 0x7fccc00f6510 2023-11-15T06:40:48.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.672+0000 7fccbf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc01066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:48.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fccbf7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc01066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48652/0 (socket says 172.21.15.135:48652) 2023-11-15T06:40:48.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fccbf7fe700 1 -- 172.21.15.135:0/2311090630 learned_addr learned my addr 172.21.15.135:0/2311090630 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:48.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fccbf7fe700 1 -- 172.21.15.135:0/2311090630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fccb4005040 con 0x7fccc00f6510 2023-11-15T06:40:48.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fccbf7fe700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc01066d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fccb4014000 tx=0x7fccb400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:48.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fcca7fff700 1 -- 172.21.15.135:0/2311090630 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb40095f0 con 0x7fccc00f6510 2023-11-15T06:40:48.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccc0106dc0 con 0x7fccc00f6510 2023-11-15T06:40:48.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fcca7fff700 1 -- 172.21.15.135:0/2311090630 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fccb4007e70 con 0x7fccc00f6510 2023-11-15T06:40:48.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fcca7fff700 1 -- 172.21.15.135:0/2311090630 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fccb4014ea0 con 0x7fccc00f6510 2023-11-15T06:40:48.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.673+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccc0107250 con 0x7fccc00f6510 2023-11-15T06:40:48.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.674+0000 7fcca7fff700 1 -- 172.21.15.135:0/2311090630 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fccb4025070 con 0x7fccc00f6510 2023-11-15T06:40:48.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.674+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccc0045ad0 con 0x7fccc00f6510 2023-11-15T06:40:48.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.674+0000 7fcca7fff700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcca802ff60 0x7fcca8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:48.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.674+0000 7fcca7fff700 1 -- 172.21.15.135:0/2311090630 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fccb4040930 con 0x7fccc00f6510 2023-11-15T06:40:48.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.675+0000 7fccbeffd700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcca802ff60 0x7fcca8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:48.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.675+0000 7fccbeffd700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcca802ff60 0x7fcca8032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fccb00096a0 tx=0x7fccb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:48.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.679+0000 7fcca7fff700 1 -- 172.21.15.135:0/2311090630 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fccb400f4e0 con 0x7fccc00f6510 2023-11-15T06:40:48.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.884+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccc0108100 con 0x7fccc00f6510 2023-11-15T06:40:48.888 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.885+0000 7fcca7fff700 1 -- 172.21.15.135:0/2311090630 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fccb400f300 con 0x7fccc00f6510 2023-11-15T06:40:48.888 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:48.888 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:48.889 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcca802ff60 msgr2=0x7fcca8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:48.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcca802ff60 0x7fcca8032420 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7fccb00096a0 tx=0x7fccb0006b40).stop 2023-11-15T06:40:48.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 msgr2=0x7fccc01066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:48.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc01066d0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7fccb4014000 tx=0x7fccb400b040).stop 2023-11-15T06:40:48.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 shutdown_connections 2023-11-15T06:40:48.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcca802ff60 0x7fcca8032420 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:48.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 --2- 172.21.15.135:0/2311090630 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fccc00f6510 0x7fccc01066d0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:48.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 >> 172.21.15.135:0/2311090630 conn(0x7fccc00f1c70 msgr2=0x7fccc00f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:48.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 shutdown_connections 2023-11-15T06:40:48.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:48.887+0000 7fccc5d1c700 1 -- 172.21.15.135:0/2311090630 wait complete. 2023-11-15T06:40:48.892 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:49.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:48 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:48.886052+0000 mon.smithi123 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.135:0/2311090630' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:50.265 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:50.265 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:50.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:48.259102+0000 mgr.smithi123.razjtn (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:51.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.439+0000 7fcb3743f700 1 -- 172.21.15.135:0/1314553132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 msgr2=0x7fcb300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:51.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.439+0000 7fcb3743f700 1 --2- 172.21.15.135:0/1314553132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb300f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fcb20009fb0 tx=0x7fcb20009b30).stop 2023-11-15T06:40:51.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.439+0000 7fcb3743f700 1 -- 172.21.15.135:0/1314553132 shutdown_connections 2023-11-15T06:40:51.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.439+0000 7fcb3743f700 1 --2- 172.21.15.135:0/1314553132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb300f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:51.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.439+0000 7fcb3743f700 1 -- 172.21.15.135:0/1314553132 >> 172.21.15.135:0/1314553132 conn(0x7fcb300f0b70 msgr2=0x7fcb300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:51.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.440+0000 7fcb3743f700 1 -- 172.21.15.135:0/1314553132 shutdown_connections 2023-11-15T06:40:51.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.440+0000 7fcb3743f700 1 -- 172.21.15.135:0/1314553132 wait complete. 2023-11-15T06:40:51.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.440+0000 7fcb3743f700 1 Processor -- start 2023-11-15T06:40:51.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.440+0000 7fcb3743f700 1 -- start start 2023-11-15T06:40:51.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.440+0000 7fcb3743f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb301066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:51.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.440+0000 7fcb3743f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb30106bd0 con 0x7fcb300f5410 2023-11-15T06:40:51.446 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb351db700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb301066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:51.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb351db700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb301066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48666/0 (socket says 172.21.15.135:48666) 2023-11-15T06:40:51.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb351db700 1 -- 172.21.15.135:0/197020298 learned_addr learned my addr 172.21.15.135:0/197020298 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:51.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb351db700 1 -- 172.21.15.135:0/197020298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb20005040 con 0x7fcb300f5410 2023-11-15T06:40:51.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb351db700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb301066e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fcb20014000 tx=0x7fcb2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:51.447 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb25ffb700 1 -- 172.21.15.135:0/197020298 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb200095f0 con 0x7fcb300f5410 2023-11-15T06:40:51.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb30106dd0 con 0x7fcb300f5410 2023-11-15T06:40:51.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb25ffb700 1 -- 172.21.15.135:0/197020298 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb20007e70 con 0x7fcb300f5410 2023-11-15T06:40:51.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb25ffb700 1 -- 172.21.15.135:0/197020298 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb20014ea0 con 0x7fcb300f5410 2023-11-15T06:40:51.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.441+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb30107260 con 0x7fcb300f5410 2023-11-15T06:40:51.448 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.442+0000 7fcb25ffb700 1 -- 172.21.15.135:0/197020298 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb20025070 con 0x7fcb300f5410 2023-11-15T06:40:51.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.442+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb30045ad0 con 0x7fcb300f5410 2023-11-15T06:40:51.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.442+0000 7fcb25ffb700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb1c02ff60 0x7fcb1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:51.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.442+0000 7fcb25ffb700 1 -- 172.21.15.135:0/197020298 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb20040c90 con 0x7fcb300f5410 2023-11-15T06:40:51.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.442+0000 7fcb349da700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb1c02ff60 0x7fcb1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:51.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.443+0000 7fcb349da700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb1c02ff60 0x7fcb1c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fcb2c0096a0 tx=0x7fcb2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:51.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.447+0000 7fcb25ffb700 1 -- 172.21.15.135:0/197020298 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb200079c0 con 0x7fcb300f5410 2023-11-15T06:40:51.655 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.652+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb300fae60 con 0x7fcb300f5410 2023-11-15T06:40:51.655 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.653+0000 7fcb25ffb700 1 -- 172.21.15.135:0/197020298 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb20012330 con 0x7fcb300f5410 2023-11-15T06:40:51.656 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:51.656 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:51.657 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb1c02ff60 msgr2=0x7fcb1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:51.657 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb1c02ff60 0x7fcb1c032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fcb2c0096a0 tx=0x7fcb2c006b40).stop 2023-11-15T06:40:51.658 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 msgr2=0x7fcb301066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:51.658 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb301066e0 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fcb20014000 tx=0x7fcb2000b040).stop 2023-11-15T06:40:51.658 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 shutdown_connections 2023-11-15T06:40:51.658 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb1c02ff60 0x7fcb1c032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:51.658 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 --2- 172.21.15.135:0/197020298 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb300f5410 0x7fcb301066e0 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:51.659 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 >> 172.21.15.135:0/197020298 conn(0x7fcb300f0b70 msgr2=0x7fcb3005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:51.659 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 shutdown_connections 2023-11-15T06:40:51.659 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:51.654+0000 7fcb3743f700 1 -- 172.21.15.135:0/197020298 wait complete. 2023-11-15T06:40:51.659 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:52.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:50.259449+0000 mgr.smithi123.razjtn (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:52.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:51.653863+0000 mon.smithi123 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.135:0/197020298' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:53.035 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:53.036 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:54.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.157+0000 7fcdf495d700 1 -- 172.21.15.135:0/431993270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 msgr2=0x7fcdf00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:54.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.157+0000 7fcdf495d700 1 --2- 172.21.15.135:0/431993270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf00f55c0 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7fcde4009fb0 tx=0x7fcde4009b30).stop 2023-11-15T06:40:54.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.157+0000 7fcdf495d700 1 -- 172.21.15.135:0/431993270 shutdown_connections 2023-11-15T06:40:54.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.157+0000 7fcdf495d700 1 --2- 172.21.15.135:0/431993270 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf00f55c0 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:54.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.157+0000 7fcdf495d700 1 -- 172.21.15.135:0/431993270 >> 172.21.15.135:0/431993270 conn(0x7fcdf00f09a0 msgr2=0x7fcdf00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:54.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.157+0000 7fcdf495d700 1 -- 172.21.15.135:0/431993270 shutdown_connections 2023-11-15T06:40:54.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.157+0000 7fcdf495d700 1 -- 172.21.15.135:0/431993270 wait complete. 2023-11-15T06:40:54.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.158+0000 7fcdf495d700 1 Processor -- start 2023-11-15T06:40:54.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.158+0000 7fcdf495d700 1 -- start start 2023-11-15T06:40:54.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.158+0000 7fcdf495d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf01053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:54.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.158+0000 7fcdf495d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcdf01058a0 con 0x7fcdf00f51a0 2023-11-15T06:40:54.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.158+0000 7fcdeed9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf01053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:54.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdeed9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf01053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48690/0 (socket says 172.21.15.135:48690) 2023-11-15T06:40:54.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdeed9d700 1 -- 172.21.15.135:0/2670705547 learned_addr learned my addr 172.21.15.135:0/2670705547 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:54.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdeed9d700 1 -- 172.21.15.135:0/2670705547 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcde4005040 con 0x7fcdf00f51a0 2023-11-15T06:40:54.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdeed9d700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf01053b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fcde4013510 tx=0x7fcde400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:54.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdd77fe700 1 -- 172.21.15.135:0/2670705547 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde4009320 con 0x7fcdf00f51a0 2023-11-15T06:40:54.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcdf0105aa0 con 0x7fcdf00f51a0 2023-11-15T06:40:54.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdd77fe700 1 -- 172.21.15.135:0/2670705547 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcde4013f00 con 0x7fcdf00f51a0 2023-11-15T06:40:54.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdd77fe700 1 -- 172.21.15.135:0/2670705547 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcde4014dc0 con 0x7fcdf00f51a0 2023-11-15T06:40:54.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.159+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcdf0105f30 con 0x7fcdf00f51a0 2023-11-15T06:40:54.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.160+0000 7fcdd77fe700 1 -- 172.21.15.135:0/2670705547 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcde401d3d0 con 0x7fcdf00f51a0 2023-11-15T06:40:54.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.160+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcdf0045ad0 con 0x7fcdf00f51a0 2023-11-15T06:40:54.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.160+0000 7fcdd77fe700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdd802ff60 0x7fcdd8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:54.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.160+0000 7fcdd77fe700 1 -- 172.21.15.135:0/2670705547 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcde4041110 con 0x7fcdf00f51a0 2023-11-15T06:40:54.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.160+0000 7fcdee59c700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdd802ff60 0x7fcdd8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:54.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.161+0000 7fcdee59c700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdd802ff60 0x7fcdd8032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fcde00096a0 tx=0x7fcde0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:54.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.165+0000 7fcdd77fe700 1 -- 172.21.15.135:0/2670705547 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcde40126f0 con 0x7fcdf00f51a0 2023-11-15T06:40:54.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.374+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcdf0106de0 con 0x7fcdf00f51a0 2023-11-15T06:40:54.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.374+0000 7fcdd77fe700 1 -- 172.21.15.135:0/2670705547 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcde4012330 con 0x7fcdf00f51a0 2023-11-15T06:40:54.378 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:54.378 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:54.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdd802ff60 msgr2=0x7fcdd8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:54.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdd802ff60 0x7fcdd8032420 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7fcde00096a0 tx=0x7fcde0006b40).stop 2023-11-15T06:40:54.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 msgr2=0x7fcdf01053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:54.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf01053b0 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7fcde4013510 tx=0x7fcde400b040).stop 2023-11-15T06:40:54.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 shutdown_connections 2023-11-15T06:40:54.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcdd802ff60 0x7fcdd8032420 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:54.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 --2- 172.21.15.135:0/2670705547 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcdf00f51a0 0x7fcdf01053b0 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:54.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 >> 172.21.15.135:0/2670705547 conn(0x7fcdf00f09a0 msgr2=0x7fcdf00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:54.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 shutdown_connections 2023-11-15T06:40:54.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:54.376+0000 7fcdf495d700 1 -- 172.21.15.135:0/2670705547 wait complete. 2023-11-15T06:40:54.381 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:54.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:52.259830+0000 mgr.smithi123.razjtn (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:55.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:54 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:54.375584+0000 mon.smithi123 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.135:0/2670705547' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:55.752 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:55.753 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:56.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:54.260201+0000 mgr.smithi123.razjtn (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:56.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 -- 172.21.15.135:0/1109147751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9800f51a0 msgr2=0x7fe9800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:56.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 --2- 172.21.15.135:0/1109147751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9800f51a0 0x7fe9800f55c0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fe974009fb0 tx=0x7fe974009b30).stop 2023-11-15T06:40:56.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 -- 172.21.15.135:0/1109147751 shutdown_connections 2023-11-15T06:40:56.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 --2- 172.21.15.135:0/1109147751 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe9800f51a0 0x7fe9800f55c0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:56.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 -- 172.21.15.135:0/1109147751 >> 172.21.15.135:0/1109147751 conn(0x7fe9800f09a0 msgr2=0x7fe9800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:56.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 -- 172.21.15.135:0/1109147751 shutdown_connections 2023-11-15T06:40:56.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 -- 172.21.15.135:0/1109147751 wait complete. 2023-11-15T06:40:56.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 Processor -- start 2023-11-15T06:40:56.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.906+0000 7fe985f02700 1 -- start start 2023-11-15T06:40:56.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.907+0000 7fe985f02700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe980106610 0x7fe980106a30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:56.912 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.907+0000 7fe985f02700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe980106f20 con 0x7fe980106610 2023-11-15T06:40:56.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.907+0000 7fe984f00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe980106610 0x7fe980106a30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:56.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.907+0000 7fe984f00700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe980106610 0x7fe980106a30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48714/0 (socket says 172.21.15.135:48714) 2023-11-15T06:40:56.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.907+0000 7fe984f00700 1 -- 172.21.15.135:0/4270674376 learned_addr learned my addr 172.21.15.135:0/4270674376 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:56.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.907+0000 7fe984f00700 1 -- 172.21.15.135:0/4270674376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe974005040 con 0x7fe980106610 2023-11-15T06:40:56.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.907+0000 7fe984f00700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe980106610 0x7fe980106a30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe974009f80 tx=0x7fe974009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:56.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.908+0000 7fe97d7fa700 1 -- 172.21.15.135:0/4270674376 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe9740136c0 con 0x7fe980106610 2023-11-15T06:40:56.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.908+0000 7fe97d7fa700 1 -- 172.21.15.135:0/4270674376 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe9740095f0 con 0x7fe980106610 2023-11-15T06:40:56.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.908+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe980107120 con 0x7fe980106610 2023-11-15T06:40:56.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.908+0000 7fe97d7fa700 1 -- 172.21.15.135:0/4270674376 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe974014420 con 0x7fe980106610 2023-11-15T06:40:56.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.908+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe980107c70 con 0x7fe980106610 2023-11-15T06:40:56.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.908+0000 7fe97d7fa700 1 -- 172.21.15.135:0/4270674376 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe974025060 con 0x7fe980106610 2023-11-15T06:40:56.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.909+0000 7fe97d7fa700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe97002ff60 0x7fe970032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:56.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.908+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe980059100 con 0x7fe980106610 2023-11-15T06:40:56.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.909+0000 7fe97d7fa700 1 -- 172.21.15.135:0/4270674376 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe974040e70 con 0x7fe980106610 2023-11-15T06:40:56.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.909+0000 7fe97ffff700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe97002ff60 0x7fe970032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:56.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.909+0000 7fe97ffff700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe97002ff60 0x7fe970032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe96c0096a0 tx=0x7fe96c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:56.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:56.913+0000 7fe97d7fa700 1 -- 172.21.15.135:0/4270674376 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe974004490 con 0x7fe980106610 2023-11-15T06:40:57.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.118+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe980045ad0 con 0x7fe980106610 2023-11-15T06:40:57.121 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.119+0000 7fe97d7fa700 1 -- 172.21.15.135:0/4270674376 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe974004490 con 0x7fe980106610 2023-11-15T06:40:57.122 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:57.122 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:57.123 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe97002ff60 msgr2=0x7fe970032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:57.124 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe97002ff60 0x7fe970032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fe96c0096a0 tx=0x7fe96c006b40).stop 2023-11-15T06:40:57.124 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe980106610 msgr2=0x7fe980106a30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:57.124 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe980106610 0x7fe980106a30 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fe974009f80 tx=0x7fe974009b30).stop 2023-11-15T06:40:57.124 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 shutdown_connections 2023-11-15T06:40:57.124 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe97002ff60 0x7fe970032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:57.125 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 --2- 172.21.15.135:0/4270674376 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe980106610 0x7fe980106a30 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:57.125 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 >> 172.21.15.135:0/4270674376 conn(0x7fe9800f09a0 msgr2=0x7fe98005fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:57.125 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 shutdown_connections 2023-11-15T06:40:57.125 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:57.120+0000 7fe985f02700 1 -- 172.21.15.135:0/4270674376 wait complete. 2023-11-15T06:40:57.125 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:40:58.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:56.260563+0000 mgr.smithi123.razjtn (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:40:58.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:57.119710+0000 mon.smithi123 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.135:0/4270674376' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:40:58.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:40:58.521 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:40:59.615 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.610+0000 7fed87c2f700 1 -- 172.21.15.135:0/4083526625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 msgr2=0x7fed800f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:59.615 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.610+0000 7fed87c2f700 1 --2- 172.21.15.135:0/4083526625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed800f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7fed70009fb0 tx=0x7fed70009b30).stop 2023-11-15T06:40:59.615 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.610+0000 7fed87c2f700 1 -- 172.21.15.135:0/4083526625 shutdown_connections 2023-11-15T06:40:59.616 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.610+0000 7fed87c2f700 1 --2- 172.21.15.135:0/4083526625 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed800f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:59.616 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.610+0000 7fed87c2f700 1 -- 172.21.15.135:0/4083526625 >> 172.21.15.135:0/4083526625 conn(0x7fed800f0b70 msgr2=0x7fed800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:59.616 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed87c2f700 1 -- 172.21.15.135:0/4083526625 shutdown_connections 2023-11-15T06:40:59.616 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed87c2f700 1 -- 172.21.15.135:0/4083526625 wait complete. 2023-11-15T06:40:59.617 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed87c2f700 1 Processor -- start 2023-11-15T06:40:59.617 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed87c2f700 1 -- start start 2023-11-15T06:40:59.617 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed87c2f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed80106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:59.617 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed87c2f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed80106c00 con 0x7fed800f5410 2023-11-15T06:40:59.617 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed859cb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed80106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:59.618 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed859cb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed80106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48180/0 (socket says 172.21.15.135:48180) 2023-11-15T06:40:59.618 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.611+0000 7fed859cb700 1 -- 172.21.15.135:0/327921482 learned_addr learned my addr 172.21.15.135:0/327921482 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:40:59.618 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.612+0000 7fed859cb700 1 -- 172.21.15.135:0/327921482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed70005040 con 0x7fed800f5410 2023-11-15T06:40:59.618 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.612+0000 7fed859cb700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed80106710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fed70014000 tx=0x7fed7000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:59.618 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.612+0000 7fed767fc700 1 -- 172.21.15.135:0/327921482 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed700095f0 con 0x7fed800f5410 2023-11-15T06:40:59.619 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.612+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed80106e00 con 0x7fed800f5410 2023-11-15T06:40:59.619 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.612+0000 7fed767fc700 1 -- 172.21.15.135:0/327921482 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed70007e70 con 0x7fed800f5410 2023-11-15T06:40:59.619 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.612+0000 7fed767fc700 1 -- 172.21.15.135:0/327921482 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed70014ea0 con 0x7fed800f5410 2023-11-15T06:40:59.619 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.612+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed80107290 con 0x7fed800f5410 2023-11-15T06:40:59.620 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.613+0000 7fed767fc700 1 -- 172.21.15.135:0/327921482 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed70025070 con 0x7fed800f5410 2023-11-15T06:40:59.621 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.613+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed80045ad0 con 0x7fed800f5410 2023-11-15T06:40:59.621 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.613+0000 7fed767fc700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6c02ff60 0x7fed6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:40:59.621 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.613+0000 7fed767fc700 1 -- 172.21.15.135:0/327921482 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed700409f0 con 0x7fed800f5410 2023-11-15T06:40:59.621 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.613+0000 7fed851ca700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6c02ff60 0x7fed6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:40:59.621 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.614+0000 7fed851ca700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6c02ff60 0x7fed6c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fed7c0096a0 tx=0x7fed7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:40:59.622 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.618+0000 7fed767fc700 1 -- 172.21.15.135:0/327921482 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed7000f4e0 con 0x7fed800f5410 2023-11-15T06:40:59.829 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.827+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed800fae60 con 0x7fed800f5410 2023-11-15T06:40:59.830 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.828+0000 7fed767fc700 1 -- 172.21.15.135:0/327921482 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed7000f300 con 0x7fed800f5410 2023-11-15T06:40:59.831 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:40:59.831 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:40:59.832 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6c02ff60 msgr2=0x7fed6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:59.832 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6c02ff60 0x7fed6c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fed7c0096a0 tx=0x7fed7c006b40).stop 2023-11-15T06:40:59.832 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 msgr2=0x7fed80106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:40:59.833 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed80106710 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7fed70014000 tx=0x7fed7000b040).stop 2023-11-15T06:40:59.833 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 shutdown_connections 2023-11-15T06:40:59.833 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6c02ff60 0x7fed6c032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:59.833 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 --2- 172.21.15.135:0/327921482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed800f5410 0x7fed80106710 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:40:59.833 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 >> 172.21.15.135:0/327921482 conn(0x7fed800f0b70 msgr2=0x7fed8005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:40:59.834 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 shutdown_connections 2023-11-15T06:40:59.834 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:40:59.829+0000 7fed87c2f700 1 -- 172.21.15.135:0/327921482 wait complete. 2023-11-15T06:40:59.834 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:00.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:40:58.260863+0000 mgr.smithi123.razjtn (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:00.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:40:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:40:59.828835+0000 mon.smithi123 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.135:0/327921482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:01.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:01.206 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:02.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.381+0000 7f669ade1700 1 -- 172.21.15.135:0/3288239490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 msgr2=0x7f66940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:02.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.381+0000 7f669ade1700 1 --2- 172.21.15.135:0/3288239490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f66940f6930 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f6688009fb0 tx=0x7f6688009b30).stop 2023-11-15T06:41:02.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.381+0000 7f669ade1700 1 -- 172.21.15.135:0/3288239490 shutdown_connections 2023-11-15T06:41:02.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.381+0000 7f669ade1700 1 --2- 172.21.15.135:0/3288239490 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f66940f6930 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:02.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.381+0000 7f669ade1700 1 -- 172.21.15.135:0/3288239490 >> 172.21.15.135:0/3288239490 conn(0x7f66940f1c70 msgr2=0x7f66940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:02.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.381+0000 7f669ade1700 1 -- 172.21.15.135:0/3288239490 shutdown_connections 2023-11-15T06:41:02.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.381+0000 7f669ade1700 1 -- 172.21.15.135:0/3288239490 wait complete. 2023-11-15T06:41:02.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.382+0000 7f669ade1700 1 Processor -- start 2023-11-15T06:41:02.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.382+0000 7f669ade1700 1 -- start start 2023-11-15T06:41:02.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.382+0000 7f669ade1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f6694106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:02.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.382+0000 7f669ade1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6694106c20 con 0x7f66940f6510 2023-11-15T06:41:02.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.382+0000 7f6698b7d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f6694106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:02.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.382+0000 7f6698b7d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f6694106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48200/0 (socket says 172.21.15.135:48200) 2023-11-15T06:41:02.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.382+0000 7f6698b7d700 1 -- 172.21.15.135:0/62418285 learned_addr learned my addr 172.21.15.135:0/62418285 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:02.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.383+0000 7f6698b7d700 1 -- 172.21.15.135:0/62418285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6688005040 con 0x7f66940f6510 2023-11-15T06:41:02.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.383+0000 7f6698b7d700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f6694106730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f6688013510 tx=0x7f668800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:02.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.383+0000 7f66917fa700 1 -- 172.21.15.135:0/62418285 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6688009320 con 0x7f66940f6510 2023-11-15T06:41:02.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.383+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6694106e20 con 0x7f66940f6510 2023-11-15T06:41:02.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.383+0000 7f66917fa700 1 -- 172.21.15.135:0/62418285 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6688013f00 con 0x7f66940f6510 2023-11-15T06:41:02.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.383+0000 7f66917fa700 1 -- 172.21.15.135:0/62418285 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6688014d90 con 0x7f66940f6510 2023-11-15T06:41:02.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.383+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66941072b0 con 0x7f66940f6510 2023-11-15T06:41:02.392 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.384+0000 7f66917fa700 1 -- 172.21.15.135:0/62418285 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66880263a0 con 0x7f66940f6510 2023-11-15T06:41:02.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.384+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6694045ad0 con 0x7f66940f6510 2023-11-15T06:41:02.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.384+0000 7f66917fa700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f668402ff60 0x7f6684032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:02.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.384+0000 7f66917fa700 1 -- 172.21.15.135:0/62418285 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66880413c0 con 0x7f66940f6510 2023-11-15T06:41:02.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.384+0000 7f6693fff700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f668402ff60 0x7f6684032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:02.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.384+0000 7f6693fff700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f668402ff60 0x7f6684032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f66800096a0 tx=0x7f6680006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:02.394 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.389+0000 7f66917fa700 1 -- 172.21.15.135:0/62418285 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6688012330 con 0x7f66940f6510 2023-11-15T06:41:02.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:00.261178+0000 mgr.smithi123.razjtn (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:02.599 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.597+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66940fbf60 con 0x7f66940f6510 2023-11-15T06:41:02.600 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.597+0000 7f66917fa700 1 -- 172.21.15.135:0/62418285 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6688012330 con 0x7f66940f6510 2023-11-15T06:41:02.601 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:02.601 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:02.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f668402ff60 msgr2=0x7f6684032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:02.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f668402ff60 0x7f6684032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f66800096a0 tx=0x7f6680006b40).stop 2023-11-15T06:41:02.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 msgr2=0x7f6694106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:02.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f6694106730 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f6688013510 tx=0x7f668800b040).stop 2023-11-15T06:41:02.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 shutdown_connections 2023-11-15T06:41:02.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f668402ff60 0x7f6684032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:02.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 --2- 172.21.15.135:0/62418285 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66940f6510 0x7f6694106730 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:02.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 >> 172.21.15.135:0/62418285 conn(0x7f66940f1c70 msgr2=0x7f66940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:02.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 shutdown_connections 2023-11-15T06:41:02.604 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:02.599+0000 7f669ade1700 1 -- 172.21.15.135:0/62418285 wait complete. 2023-11-15T06:41:02.604 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:02.261545+0000 mgr.smithi123.razjtn (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:02.598594+0000 mon.smithi123 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.135:0/62418285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:04.024 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:04.025 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:05.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.204+0000 7fb139c89700 1 -- 172.21.15.135:0/4263302997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 msgr2=0x7fb1340f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:05.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.204+0000 7fb139c89700 1 --2- 172.21.15.135:0/4263302997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1340f90f0 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7fb124009fb0 tx=0x7fb124009b30).stop 2023-11-15T06:41:05.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.204+0000 7fb139c89700 1 -- 172.21.15.135:0/4263302997 shutdown_connections 2023-11-15T06:41:05.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.204+0000 7fb139c89700 1 --2- 172.21.15.135:0/4263302997 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1340f90f0 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:05.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.204+0000 7fb139c89700 1 -- 172.21.15.135:0/4263302997 >> 172.21.15.135:0/4263302997 conn(0x7fb1340f0b70 msgr2=0x7fb1340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:05.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.204+0000 7fb139c89700 1 -- 172.21.15.135:0/4263302997 shutdown_connections 2023-11-15T06:41:05.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.204+0000 7fb139c89700 1 -- 172.21.15.135:0/4263302997 wait complete. 2023-11-15T06:41:05.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.205+0000 7fb139c89700 1 Processor -- start 2023-11-15T06:41:05.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.205+0000 7fb139c89700 1 -- start start 2023-11-15T06:41:05.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.205+0000 7fb139c89700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1341066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:05.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.205+0000 7fb139c89700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb134106bd0 con 0x7fb134060280 2023-11-15T06:41:05.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb1337fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1341066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:05.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb1337fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1341066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48210/0 (socket says 172.21.15.135:48210) 2023-11-15T06:41:05.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb1337fe700 1 -- 172.21.15.135:0/3475472939 learned_addr learned my addr 172.21.15.135:0/3475472939 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:05.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb1337fe700 1 -- 172.21.15.135:0/3475472939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb124005040 con 0x7fb134060280 2023-11-15T06:41:05.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb1337fe700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1341066e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb124013510 tx=0x7fb12400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:05.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb11bfff700 1 -- 172.21.15.135:0/3475472939 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb124009320 con 0x7fb134060280 2023-11-15T06:41:05.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb11bfff700 1 -- 172.21.15.135:0/3475472939 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb124013f00 con 0x7fb134060280 2023-11-15T06:41:05.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb11bfff700 1 -- 172.21.15.135:0/3475472939 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb124014dc0 con 0x7fb134060280 2023-11-15T06:41:05.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb134106dd0 con 0x7fb134060280 2023-11-15T06:41:05.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.206+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb134107260 con 0x7fb134060280 2023-11-15T06:41:05.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.207+0000 7fb11bfff700 1 -- 172.21.15.135:0/3475472939 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1240095f0 con 0x7fb134060280 2023-11-15T06:41:05.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.207+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb134045ad0 con 0x7fb134060280 2023-11-15T06:41:05.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.207+0000 7fb11bfff700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb11c02ffb0 0x7fb11c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:05.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.207+0000 7fb11bfff700 1 -- 172.21.15.135:0/3475472939 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb124040ff0 con 0x7fb134060280 2023-11-15T06:41:05.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.208+0000 7fb132ffd700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb11c02ffb0 0x7fb11c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:05.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.208+0000 7fb132ffd700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb11c02ffb0 0x7fb11c032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb1280096a0 tx=0x7fb128006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:05.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.212+0000 7fb11bfff700 1 -- 172.21.15.135:0/3475472939 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb12406d2b0 con 0x7fb134060280 2023-11-15T06:41:05.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.419+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb134108110 con 0x7fb134060280 2023-11-15T06:41:05.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.420+0000 7fb11bfff700 1 -- 172.21.15.135:0/3475472939 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb124014420 con 0x7fb134060280 2023-11-15T06:41:05.423 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:05.424 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:05.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb11c02ffb0 msgr2=0x7fb11c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:05.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb11c02ffb0 0x7fb11c032470 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fb1280096a0 tx=0x7fb128006b40).stop 2023-11-15T06:41:05.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 msgr2=0x7fb1341066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:05.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1341066e0 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7fb124013510 tx=0x7fb12400b040).stop 2023-11-15T06:41:05.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 shutdown_connections 2023-11-15T06:41:05.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb11c02ffb0 0x7fb11c032470 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:05.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 --2- 172.21.15.135:0/3475472939 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb134060280 0x7fb1341066e0 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:05.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 >> 172.21.15.135:0/3475472939 conn(0x7fb1340f0b70 msgr2=0x7fb1340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:05.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 shutdown_connections 2023-11-15T06:41:05.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:05.421+0000 7fb139c89700 1 -- 172.21.15.135:0/3475472939 wait complete. 2023-11-15T06:41:05.428 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:05.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:04.261918+0000 mgr.smithi123.razjtn (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:05.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:05.420720+0000 mon.smithi123 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.135:0/3475472939' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:06.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:06.803 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:07.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:06.262245+0000 mgr.smithi123.razjtn (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:07.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.988+0000 7eff2058f700 1 -- 172.21.15.135:0/1094405789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 msgr2=0x7eff180f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:07.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.988+0000 7eff2058f700 1 --2- 172.21.15.135:0/1094405789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff180f6930 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7eff0c009fb0 tx=0x7eff0c009b30).stop 2023-11-15T06:41:07.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.988+0000 7eff2058f700 1 -- 172.21.15.135:0/1094405789 shutdown_connections 2023-11-15T06:41:07.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.988+0000 7eff2058f700 1 --2- 172.21.15.135:0/1094405789 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff180f6930 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:07.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.988+0000 7eff2058f700 1 -- 172.21.15.135:0/1094405789 >> 172.21.15.135:0/1094405789 conn(0x7eff180f1c70 msgr2=0x7eff180f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:07.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.988+0000 7eff2058f700 1 -- 172.21.15.135:0/1094405789 shutdown_connections 2023-11-15T06:41:07.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.988+0000 7eff2058f700 1 -- 172.21.15.135:0/1094405789 wait complete. 2023-11-15T06:41:07.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff2058f700 1 Processor -- start 2023-11-15T06:41:07.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff2058f700 1 -- start start 2023-11-15T06:41:07.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff2058f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff18106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:07.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff2058f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7eff18106c20 con 0x7eff180f6510 2023-11-15T06:41:07.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff1e32b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff18106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:07.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff1e32b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff18106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48224/0 (socket says 172.21.15.135:48224) 2023-11-15T06:41:07.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff1e32b700 1 -- 172.21.15.135:0/640473926 learned_addr learned my addr 172.21.15.135:0/640473926 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:07.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff1e32b700 1 -- 172.21.15.135:0/640473926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7eff0c005040 con 0x7eff180f6510 2023-11-15T06:41:07.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.989+0000 7eff1e32b700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff18106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7eff0c013510 tx=0x7eff0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:07.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.990+0000 7eff0affd700 1 -- 172.21.15.135:0/640473926 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff0c009320 con 0x7eff180f6510 2023-11-15T06:41:07.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.990+0000 7eff0affd700 1 -- 172.21.15.135:0/640473926 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7eff0c013f00 con 0x7eff180f6510 2023-11-15T06:41:07.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.990+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7eff18106e20 con 0x7eff180f6510 2023-11-15T06:41:07.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.990+0000 7eff0affd700 1 -- 172.21.15.135:0/640473926 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7eff0c014d90 con 0x7eff180f6510 2023-11-15T06:41:07.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.990+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7eff181072b0 con 0x7eff180f6510 2023-11-15T06:41:07.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.990+0000 7eff0affd700 1 -- 172.21.15.135:0/640473926 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7eff0c0263a0 con 0x7eff180f6510 2023-11-15T06:41:07.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.991+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7eff18045ad0 con 0x7eff180f6510 2023-11-15T06:41:07.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.991+0000 7eff0affd700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7eff0402ffb0 0x7eff04032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:07.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.991+0000 7eff0affd700 1 -- 172.21.15.135:0/640473926 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7eff0c040ff0 con 0x7eff180f6510 2023-11-15T06:41:07.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.991+0000 7eff1db2a700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7eff0402ffb0 0x7eff04032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:07.998 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.991+0000 7eff1db2a700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7eff0402ffb0 0x7eff04032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7eff140096a0 tx=0x7eff14006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:07.999 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:07.996+0000 7eff0affd700 1 -- 172.21.15.135:0/640473926 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7eff0c06d2b0 con 0x7eff180f6510 2023-11-15T06:41:08.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.200+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7eff180fbf60 con 0x7eff180f6510 2023-11-15T06:41:08.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.201+0000 7eff0affd700 1 -- 172.21.15.135:0/640473926 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7eff0c0095f0 con 0x7eff180f6510 2023-11-15T06:41:08.204 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:08.204 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:08.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7eff0402ffb0 msgr2=0x7eff04032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:08.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7eff0402ffb0 0x7eff04032470 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7eff140096a0 tx=0x7eff14006b40).stop 2023-11-15T06:41:08.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 msgr2=0x7eff18106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:08.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff18106730 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7eff0c013510 tx=0x7eff0c00b040).stop 2023-11-15T06:41:08.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 shutdown_connections 2023-11-15T06:41:08.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7eff0402ffb0 0x7eff04032470 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:08.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 --2- 172.21.15.135:0/640473926 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7eff180f6510 0x7eff18106730 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:08.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 >> 172.21.15.135:0/640473926 conn(0x7eff180f1c70 msgr2=0x7eff180f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:08.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 shutdown_connections 2023-11-15T06:41:08.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:08.203+0000 7eff2058f700 1 -- 172.21.15.135:0/640473926 wait complete. 2023-11-15T06:41:08.208 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:08.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:08 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:08.202014+0000 mon.smithi123 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.135:0/640473926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:09.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:09.593 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:08.262584+0000 mgr.smithi123.razjtn (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:10.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 -- 172.21.15.135:0/2597924488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 msgr2=0x7f72a00f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:10.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 --2- 172.21.15.135:0/2597924488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a00f8f20 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f7290009fb0 tx=0x7f7290009b30).stop 2023-11-15T06:41:10.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 -- 172.21.15.135:0/2597924488 shutdown_connections 2023-11-15T06:41:10.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 --2- 172.21.15.135:0/2597924488 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a00f8f20 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:10.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 -- 172.21.15.135:0/2597924488 >> 172.21.15.135:0/2597924488 conn(0x7f72a00f09a0 msgr2=0x7f72a00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:10.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 -- 172.21.15.135:0/2597924488 shutdown_connections 2023-11-15T06:41:10.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 -- 172.21.15.135:0/2597924488 wait complete. 2023-11-15T06:41:10.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 Processor -- start 2023-11-15T06:41:10.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 -- start start 2023-11-15T06:41:10.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a0106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:10.735 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.727+0000 7f729f59e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72a01069f0 con 0x7f72a00601e0 2023-11-15T06:41:10.735 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f729e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a0106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:10.735 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f729e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a0106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:39992/0 (socket says 172.21.15.135:39992) 2023-11-15T06:41:10.735 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f729e59c700 1 -- 172.21.15.135:0/1607958785 learned_addr learned my addr 172.21.15.135:0/1607958785 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:10.735 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f729e59c700 1 -- 172.21.15.135:0/1607958785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7290005040 con 0x7f72a00601e0 2023-11-15T06:41:10.736 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f729e59c700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a0106500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7290013510 tx=0x7f729000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:10.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f728effd700 1 -- 172.21.15.135:0/1607958785 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7290009320 con 0x7f72a00601e0 2023-11-15T06:41:10.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72a0106bf0 con 0x7f72a00601e0 2023-11-15T06:41:10.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f728effd700 1 -- 172.21.15.135:0/1607958785 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7290013f00 con 0x7f72a00601e0 2023-11-15T06:41:10.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f728effd700 1 -- 172.21.15.135:0/1607958785 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7290014d90 con 0x7f72a00601e0 2023-11-15T06:41:10.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.728+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72a0107080 con 0x7f72a00601e0 2023-11-15T06:41:10.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.729+0000 7f728effd700 1 -- 172.21.15.135:0/1607958785 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72900095f0 con 0x7f72a00601e0 2023-11-15T06:41:10.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.729+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72a0045ad0 con 0x7f72a00601e0 2023-11-15T06:41:10.738 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.729+0000 7f728effd700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f728802ff60 0x7f7288032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:10.739 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.729+0000 7f728effd700 1 -- 172.21.15.135:0/1607958785 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72900410a0 con 0x7f72a00601e0 2023-11-15T06:41:10.739 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.729+0000 7f729dd9b700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f728802ff60 0x7f7288032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:10.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.730+0000 7f729dd9b700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f728802ff60 0x7f7288032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f72940096a0 tx=0x7f7294006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:10.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.734+0000 7f728effd700 1 -- 172.21.15.135:0/1607958785 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f729001d7f0 con 0x7f72a00601e0 2023-11-15T06:41:10.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.940+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f72a0107f30 con 0x7f72a00601e0 2023-11-15T06:41:10.944 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.941+0000 7f728effd700 1 -- 172.21.15.135:0/1607958785 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7290010230 con 0x7f72a00601e0 2023-11-15T06:41:10.944 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:10.945 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:10.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f728802ff60 msgr2=0x7f7288032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:10.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f728802ff60 0x7f7288032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f72940096a0 tx=0x7f7294006b40).stop 2023-11-15T06:41:10.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 msgr2=0x7f72a0106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:10.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a0106500 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f7290013510 tx=0x7f729000b040).stop 2023-11-15T06:41:10.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 shutdown_connections 2023-11-15T06:41:10.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f728802ff60 0x7f7288032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:10.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 --2- 172.21.15.135:0/1607958785 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f72a00601e0 0x7f72a0106500 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:10.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 >> 172.21.15.135:0/1607958785 conn(0x7f72a00f09a0 msgr2=0x7f72a00f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:10.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 shutdown_connections 2023-11-15T06:41:10.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:10.943+0000 7f729f59e700 1 -- 172.21.15.135:0/1607958785 wait complete. 2023-11-15T06:41:10.948 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:11.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:10.262858+0000 mgr.smithi123.razjtn (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:11.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:10.942349+0000 mon.smithi123 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.135:0/1607958785' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:12.326 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:12.385 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:13.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 -- 172.21.15.135:0/957585572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 msgr2=0x7f14700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:13.499 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 --2- 172.21.15.135:0/957585572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f14700f6930 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f1464009fb0 tx=0x7f1464009b30).stop 2023-11-15T06:41:13.499 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 -- 172.21.15.135:0/957585572 shutdown_connections 2023-11-15T06:41:13.499 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 --2- 172.21.15.135:0/957585572 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f14700f6930 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:13.499 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 -- 172.21.15.135:0/957585572 >> 172.21.15.135:0/957585572 conn(0x7f14700f1c70 msgr2=0x7f14700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:13.500 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 -- 172.21.15.135:0/957585572 shutdown_connections 2023-11-15T06:41:13.500 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 -- 172.21.15.135:0/957585572 wait complete. 2023-11-15T06:41:13.500 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 Processor -- start 2023-11-15T06:41:13.500 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.494+0000 7f1475baf700 1 -- start start 2023-11-15T06:41:13.500 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.495+0000 7f1475baf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f1470106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:13.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.495+0000 7f1475baf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1470106c20 con 0x7f14700f6510 2023-11-15T06:41:13.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.495+0000 7f146f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f1470106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:13.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.495+0000 7f146f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f1470106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:40010/0 (socket says 172.21.15.135:40010) 2023-11-15T06:41:13.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.495+0000 7f146f7fe700 1 -- 172.21.15.135:0/1942978506 learned_addr learned my addr 172.21.15.135:0/1942978506 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:13.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.495+0000 7f146f7fe700 1 -- 172.21.15.135:0/1942978506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1464005040 con 0x7f14700f6510 2023-11-15T06:41:13.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.495+0000 7f146f7fe700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f1470106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1464013510 tx=0x7f146400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:13.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.496+0000 7f1457fff700 1 -- 172.21.15.135:0/1942978506 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1464009320 con 0x7f14700f6510 2023-11-15T06:41:13.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.496+0000 7f1457fff700 1 -- 172.21.15.135:0/1942978506 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1464013f00 con 0x7f14700f6510 2023-11-15T06:41:13.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.496+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1470106e20 con 0x7f14700f6510 2023-11-15T06:41:13.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.496+0000 7f1457fff700 1 -- 172.21.15.135:0/1942978506 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1464014d90 con 0x7f14700f6510 2023-11-15T06:41:13.503 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.496+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f14701072b0 con 0x7f14700f6510 2023-11-15T06:41:13.504 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.496+0000 7f1457fff700 1 -- 172.21.15.135:0/1942978506 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1464014420 con 0x7f14700f6510 2023-11-15T06:41:13.504 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.496+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1470045ad0 con 0x7f14700f6510 2023-11-15T06:41:13.504 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.497+0000 7f1457fff700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f145802ff60 0x7f1458032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:13.504 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.497+0000 7f1457fff700 1 -- 172.21.15.135:0/1942978506 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f14640410c0 con 0x7f14700f6510 2023-11-15T06:41:13.504 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.497+0000 7f146effd700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f145802ff60 0x7f1458032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:13.505 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.497+0000 7f146effd700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f145802ff60 0x7f1458032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f14600096a0 tx=0x7f1460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:13.505 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.502+0000 7f1457fff700 1 -- 172.21.15.135:0/1942978506 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f14640095f0 con 0x7f14700f6510 2023-11-15T06:41:13.711 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.708+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f14700fbf60 con 0x7f14700f6510 2023-11-15T06:41:13.711 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.709+0000 7f1457fff700 1 -- 172.21.15.135:0/1942978506 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f146401db70 con 0x7f14700f6510 2023-11-15T06:41:13.712 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:13.712 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:13.714 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f145802ff60 msgr2=0x7f1458032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:13.714 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f145802ff60 0x7f1458032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f14600096a0 tx=0x7f1460006b40).stop 2023-11-15T06:41:13.714 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 msgr2=0x7f1470106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:13.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f1470106730 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1464013510 tx=0x7f146400b040).stop 2023-11-15T06:41:13.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 shutdown_connections 2023-11-15T06:41:13.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f145802ff60 0x7f1458032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:13.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 --2- 172.21.15.135:0/1942978506 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f14700f6510 0x7f1470106730 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:13.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.710+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 >> 172.21.15.135:0/1942978506 conn(0x7f14700f1c70 msgr2=0x7f14700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:13.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.711+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 shutdown_connections 2023-11-15T06:41:13.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:13.711+0000 7f1475baf700 1 -- 172.21.15.135:0/1942978506 wait complete. 2023-11-15T06:41:13.716 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:13.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:12.263189+0000 mgr.smithi123.razjtn (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:14.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:14 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:13.710004+0000 mon.smithi123 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.135:0/1942978506' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:15.096 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:15.097 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:15.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:14.263542+0000 mgr.smithi123.razjtn (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:16.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 -- 172.21.15.135:0/3230404838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 msgr2=0x7f60fc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:16.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 --2- 172.21.15.135:0/3230404838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc0f55c0 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7f60ec009fb0 tx=0x7f60ec009b30).stop 2023-11-15T06:41:16.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 -- 172.21.15.135:0/3230404838 shutdown_connections 2023-11-15T06:41:16.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 --2- 172.21.15.135:0/3230404838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc0f55c0 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:16.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 -- 172.21.15.135:0/3230404838 >> 172.21.15.135:0/3230404838 conn(0x7f60fc0f09a0 msgr2=0x7f60fc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:16.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 -- 172.21.15.135:0/3230404838 shutdown_connections 2023-11-15T06:41:16.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 -- 172.21.15.135:0/3230404838 wait complete. 2023-11-15T06:41:16.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 Processor -- start 2023-11-15T06:41:16.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 -- start start 2023-11-15T06:41:16.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc1091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:16.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.200+0000 7f610290f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60fc060240 con 0x7f60fc0f51a0 2023-11-15T06:41:16.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f610190d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc1091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:16.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f610190d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc1091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:40020/0 (socket says 172.21.15.135:40020) 2023-11-15T06:41:16.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f610190d700 1 -- 172.21.15.135:0/221844269 learned_addr learned my addr 172.21.15.135:0/221844269 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:16.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f610190d700 1 -- 172.21.15.135:0/221844269 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ec005040 con 0x7f60fc0f51a0 2023-11-15T06:41:16.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f610190d700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc1091b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f60ec013510 tx=0x7f60ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:16.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f60f27fc700 1 -- 172.21.15.135:0/221844269 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ec009320 con 0x7f60fc0f51a0 2023-11-15T06:41:16.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60fc060440 con 0x7f60fc0f51a0 2023-11-15T06:41:16.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f60f27fc700 1 -- 172.21.15.135:0/221844269 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60ec013f00 con 0x7f60fc0f51a0 2023-11-15T06:41:16.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f60f27fc700 1 -- 172.21.15.135:0/221844269 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ec014dc0 con 0x7f60fc0f51a0 2023-11-15T06:41:16.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.201+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60fc109860 con 0x7f60fc0f51a0 2023-11-15T06:41:16.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.202+0000 7f60f27fc700 1 -- 172.21.15.135:0/221844269 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60ec014420 con 0x7f60fc0f51a0 2023-11-15T06:41:16.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.202+0000 7f60f27fc700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f60e802ff60 0x7f60e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:16.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.202+0000 7f60f27fc700 1 -- 172.21.15.135:0/221844269 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60ec041050 con 0x7f60fc0f51a0 2023-11-15T06:41:16.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.202+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60fc045ad0 con 0x7f60fc0f51a0 2023-11-15T06:41:16.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.202+0000 7f610110c700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f60e802ff60 0x7f60e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:16.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.203+0000 7f610110c700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f60e802ff60 0x7f60e8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f60f80096a0 tx=0x7f60f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:16.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.207+0000 7f60f27fc700 1 -- 172.21.15.135:0/221844269 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60ec0095f0 con 0x7f60fc0f51a0 2023-11-15T06:41:16.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.413+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60fc0fabe0 con 0x7f60fc0f51a0 2023-11-15T06:41:16.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.414+0000 7f60f27fc700 1 -- 172.21.15.135:0/221844269 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f60ec02b020 con 0x7f60fc0f51a0 2023-11-15T06:41:16.417 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:16.417 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:16.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f60e802ff60 msgr2=0x7f60e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:16.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f60e802ff60 0x7f60e8032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7f60f80096a0 tx=0x7f60f8006b40).stop 2023-11-15T06:41:16.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 msgr2=0x7f60fc1091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:16.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc1091b0 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7f60ec013510 tx=0x7f60ec00b040).stop 2023-11-15T06:41:16.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 shutdown_connections 2023-11-15T06:41:16.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f60e802ff60 0x7f60e8032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:16.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 --2- 172.21.15.135:0/221844269 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60fc0f51a0 0x7f60fc1091b0 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:16.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 >> 172.21.15.135:0/221844269 conn(0x7f60fc0f09a0 msgr2=0x7f60fc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:16.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 shutdown_connections 2023-11-15T06:41:16.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:16.416+0000 7f610290f700 1 -- 172.21.15.135:0/221844269 wait complete. 2023-11-15T06:41:16.421 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:16.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:16 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:16.415159+0000 mon.smithi123 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.135:0/221844269' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:17.805 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:17.805 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:17.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:16.263904+0000 mgr.smithi123.razjtn (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:18.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 -- 172.21.15.135:0/2030868748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 msgr2=0x7f736c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:18.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 --2- 172.21.15.135:0/2030868748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c0f55c0 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f7368009fb0 tx=0x7f7368009b30).stop 2023-11-15T06:41:18.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 -- 172.21.15.135:0/2030868748 shutdown_connections 2023-11-15T06:41:18.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 --2- 172.21.15.135:0/2030868748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c0f55c0 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:18.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 -- 172.21.15.135:0/2030868748 >> 172.21.15.135:0/2030868748 conn(0x7f736c0f09a0 msgr2=0x7f736c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:18.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 -- 172.21.15.135:0/2030868748 shutdown_connections 2023-11-15T06:41:18.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 -- 172.21.15.135:0/2030868748 wait complete. 2023-11-15T06:41:18.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 Processor -- start 2023-11-15T06:41:18.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.967+0000 7f73723a4700 1 -- start start 2023-11-15T06:41:18.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.968+0000 7f73723a4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:18.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.968+0000 7f73723a4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f736c1058a0 con 0x7f736c0f51a0 2023-11-15T06:41:18.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.968+0000 7f73713a2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:18.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.968+0000 7f73713a2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:47272/0 (socket says 172.21.15.135:47272) 2023-11-15T06:41:18.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.968+0000 7f73713a2700 1 -- 172.21.15.135:0/766018584 learned_addr learned my addr 172.21.15.135:0/766018584 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:18.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.968+0000 7f73713a2700 1 -- 172.21.15.135:0/766018584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7368005040 con 0x7f736c0f51a0 2023-11-15T06:41:18.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.968+0000 7f73713a2700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c1053b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7368013510 tx=0x7f736800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:18.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.969+0000 7f7361ffb700 1 -- 172.21.15.135:0/766018584 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7368009320 con 0x7f736c0f51a0 2023-11-15T06:41:18.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.969+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f736c105aa0 con 0x7f736c0f51a0 2023-11-15T06:41:18.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.969+0000 7f7361ffb700 1 -- 172.21.15.135:0/766018584 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7368013f00 con 0x7f736c0f51a0 2023-11-15T06:41:18.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.969+0000 7f7361ffb700 1 -- 172.21.15.135:0/766018584 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7368014dc0 con 0x7f736c0f51a0 2023-11-15T06:41:18.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.969+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f736c105f30 con 0x7f736c0f51a0 2023-11-15T06:41:18.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.970+0000 7f7361ffb700 1 -- 172.21.15.135:0/766018584 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73680095f0 con 0x7f736c0f51a0 2023-11-15T06:41:18.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.970+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f736c045ad0 con 0x7f736c0f51a0 2023-11-15T06:41:18.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.970+0000 7f7361ffb700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f735802ff60 0x7f7358032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:18.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.970+0000 7f7361ffb700 1 -- 172.21.15.135:0/766018584 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7368042de0 con 0x7f736c0f51a0 2023-11-15T06:41:18.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.970+0000 7f7370ba1700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f735802ff60 0x7f7358032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:18.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.970+0000 7f7370ba1700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f735802ff60 0x7f7358032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f735c0096a0 tx=0x7f735c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:18.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:18.974+0000 7f7361ffb700 1 -- 172.21.15.135:0/766018584 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7368014420 con 0x7f736c0f51a0 2023-11-15T06:41:19.183 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.180+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f736c106de0 con 0x7f736c0f51a0 2023-11-15T06:41:19.184 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.181+0000 7f7361ffb700 1 -- 172.21.15.135:0/766018584 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f736801f7f0 con 0x7f736c0f51a0 2023-11-15T06:41:19.184 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:19.184 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:19.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f735802ff60 msgr2=0x7f7358032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:19.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f735802ff60 0x7f7358032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f735c0096a0 tx=0x7f735c006b40).stop 2023-11-15T06:41:19.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 msgr2=0x7f736c1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:19.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c1053b0 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7368013510 tx=0x7f736800b040).stop 2023-11-15T06:41:19.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 shutdown_connections 2023-11-15T06:41:19.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f735802ff60 0x7f7358032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:19.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 --2- 172.21.15.135:0/766018584 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f736c0f51a0 0x7f736c1053b0 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:19.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 >> 172.21.15.135:0/766018584 conn(0x7f736c0f09a0 msgr2=0x7f736c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:19.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 shutdown_connections 2023-11-15T06:41:19.188 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:19.183+0000 7f73723a4700 1 -- 172.21.15.135:0/766018584 wait complete. 2023-11-15T06:41:19.188 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:19.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:18.264250+0000 mgr.smithi123.razjtn (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:19.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:19.182120+0000 mon.smithi123 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.135:0/766018584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:20.572 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:20.572 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:21.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 -- 172.21.15.135:0/3734181226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 msgr2=0x7f93f00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:21.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 --2- 172.21.15.135:0/3734181226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00f55c0 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7f93e0009fb0 tx=0x7f93e0009b30).stop 2023-11-15T06:41:21.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 -- 172.21.15.135:0/3734181226 shutdown_connections 2023-11-15T06:41:21.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 --2- 172.21.15.135:0/3734181226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00f55c0 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:21.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 -- 172.21.15.135:0/3734181226 >> 172.21.15.135:0/3734181226 conn(0x7f93f00f09a0 msgr2=0x7f93f00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:21.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 -- 172.21.15.135:0/3734181226 shutdown_connections 2023-11-15T06:41:21.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 -- 172.21.15.135:0/3734181226 wait complete. 2023-11-15T06:41:21.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 Processor -- start 2023-11-15T06:41:21.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 -- start start 2023-11-15T06:41:21.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.691+0000 7f93f71c3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00fd0b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:21.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f71c3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93f00fb5a0 con 0x7f93f00f51a0 2023-11-15T06:41:21.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f61c1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00fd0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:21.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f61c1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00fd0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:47294/0 (socket says 172.21.15.135:47294) 2023-11-15T06:41:21.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f61c1700 1 -- 172.21.15.135:0/2447182647 learned_addr learned my addr 172.21.15.135:0/2447182647 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:21.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f61c1700 1 -- 172.21.15.135:0/2447182647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f93e0005040 con 0x7f93f00f51a0 2023-11-15T06:41:21.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f61c1700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00fd0b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f93e0013510 tx=0x7f93e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:21.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93e6ffd700 1 -- 172.21.15.135:0/2447182647 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93e0009320 con 0x7f93f00f51a0 2023-11-15T06:41:21.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93e6ffd700 1 -- 172.21.15.135:0/2447182647 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f93e0013f00 con 0x7f93f00f51a0 2023-11-15T06:41:21.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93f00fb7a0 con 0x7f93f00f51a0 2023-11-15T06:41:21.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93e6ffd700 1 -- 172.21.15.135:0/2447182647 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f93e0014dc0 con 0x7f93f00f51a0 2023-11-15T06:41:21.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.692+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93f00fbc30 con 0x7f93f00f51a0 2023-11-15T06:41:21.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.693+0000 7f93e6ffd700 1 -- 172.21.15.135:0/2447182647 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93e00095f0 con 0x7f93f00f51a0 2023-11-15T06:41:21.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.693+0000 7f93e6ffd700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f93dc02ff60 0x7f93dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:21.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.694+0000 7f93e6ffd700 1 -- 172.21.15.135:0/2447182647 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f93e0041000 con 0x7f93f00f51a0 2023-11-15T06:41:21.701 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.694+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93f0059100 con 0x7f93f00f51a0 2023-11-15T06:41:21.701 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.694+0000 7f93f59c0700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f93dc02ff60 0x7f93dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:21.701 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.694+0000 7f93f59c0700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f93dc02ff60 0x7f93dc032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f93ec0096a0 tx=0x7f93ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:21.702 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.698+0000 7f93e6ffd700 1 -- 172.21.15.135:0/2447182647 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f93e001c790 con 0x7f93f00f51a0 2023-11-15T06:41:21.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:20.264604+0000 mgr.smithi123.razjtn (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:21.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.906+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93f00fc2c0 con 0x7f93f00f51a0 2023-11-15T06:41:21.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.907+0000 7f93e6ffd700 1 -- 172.21.15.135:0/2447182647 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f93e002b020 con 0x7f93f00f51a0 2023-11-15T06:41:21.909 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:21.910 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:21.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f93dc02ff60 msgr2=0x7f93dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:21.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f93dc02ff60 0x7f93dc032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7f93ec0096a0 tx=0x7f93ec006b40).stop 2023-11-15T06:41:21.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 msgr2=0x7f93f00fd0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:21.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00fd0b0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7f93e0013510 tx=0x7f93e000b040).stop 2023-11-15T06:41:21.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 shutdown_connections 2023-11-15T06:41:21.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f93dc02ff60 0x7f93dc032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:21.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 --2- 172.21.15.135:0/2447182647 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f93f00f51a0 0x7f93f00fd0b0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:21.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 >> 172.21.15.135:0/2447182647 conn(0x7f93f00f09a0 msgr2=0x7f93f005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:21.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 shutdown_connections 2023-11-15T06:41:21.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:21.908+0000 7f93f71c3700 1 -- 172.21.15.135:0/2447182647 wait complete. 2023-11-15T06:41:21.916 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:22.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:22 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:21.907816+0000 mon.smithi123 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.135:0/2447182647' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:23.310 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:23.311 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:23.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:22.264901+0000 mgr.smithi123.razjtn (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:24.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.452+0000 7fa590d7d700 1 -- 172.21.15.135:0/3028158504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 msgr2=0x7fa58c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:24.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.452+0000 7fa590d7d700 1 --2- 172.21.15.135:0/3028158504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c0f5830 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7fa580009fb0 tx=0x7fa580009b30).stop 2023-11-15T06:41:24.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.452+0000 7fa590d7d700 1 -- 172.21.15.135:0/3028158504 shutdown_connections 2023-11-15T06:41:24.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.452+0000 7fa590d7d700 1 --2- 172.21.15.135:0/3028158504 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c0f5830 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:24.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.452+0000 7fa590d7d700 1 -- 172.21.15.135:0/3028158504 >> 172.21.15.135:0/3028158504 conn(0x7fa58c0f0b70 msgr2=0x7fa58c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:24.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.452+0000 7fa590d7d700 1 -- 172.21.15.135:0/3028158504 shutdown_connections 2023-11-15T06:41:24.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.452+0000 7fa590d7d700 1 -- 172.21.15.135:0/3028158504 wait complete. 2023-11-15T06:41:24.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa590d7d700 1 Processor -- start 2023-11-15T06:41:24.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa590d7d700 1 -- start start 2023-11-15T06:41:24.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa590d7d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:24.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa590d7d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa58c060220 con 0x7fa58c0f5410 2023-11-15T06:41:24.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa58a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:24.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa58a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:47312/0 (socket says 172.21.15.135:47312) 2023-11-15T06:41:24.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa58a59c700 1 -- 172.21.15.135:0/3304800992 learned_addr learned my addr 172.21.15.135:0/3304800992 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:24.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa58a59c700 1 -- 172.21.15.135:0/3304800992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa580005040 con 0x7fa58c0f5410 2023-11-15T06:41:24.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.453+0000 7fa58a59c700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c109450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fa580013510 tx=0x7fa58000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:24.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa57affd700 1 -- 172.21.15.135:0/3304800992 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa580009320 con 0x7fa58c0f5410 2023-11-15T06:41:24.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa58c060420 con 0x7fa58c0f5410 2023-11-15T06:41:24.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa57affd700 1 -- 172.21.15.135:0/3304800992 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa580013f00 con 0x7fa58c0f5410 2023-11-15T06:41:24.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa57affd700 1 -- 172.21.15.135:0/3304800992 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa580014d90 con 0x7fa58c0f5410 2023-11-15T06:41:24.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa58c109b00 con 0x7fa58c0f5410 2023-11-15T06:41:24.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa57affd700 1 -- 172.21.15.135:0/3304800992 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa580014420 con 0x7fa58c0f5410 2023-11-15T06:41:24.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa57affd700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa57402ff60 0x7fa574032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:24.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa57affd700 1 -- 172.21.15.135:0/3304800992 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5800410c0 con 0x7fa58c0f5410 2023-11-15T06:41:24.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.454+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa58c059100 con 0x7fa58c0f5410 2023-11-15T06:41:24.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.455+0000 7fa589d9b700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa57402ff60 0x7fa574032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:24.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.455+0000 7fa589d9b700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa57402ff60 0x7fa574032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa57c0096a0 tx=0x7fa57c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:24.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.459+0000 7fa57affd700 1 -- 172.21.15.135:0/3304800992 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5800095f0 con 0x7fa58c0f5410 2023-11-15T06:41:24.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.664+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa58c045ad0 con 0x7fa58c0f5410 2023-11-15T06:41:24.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.665+0000 7fa57affd700 1 -- 172.21.15.135:0/3304800992 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa58c045ad0 con 0x7fa58c0f5410 2023-11-15T06:41:24.668 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:24.669 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:24.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa57402ff60 msgr2=0x7fa574032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:24.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa57402ff60 0x7fa574032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fa57c0096a0 tx=0x7fa57c006b40).stop 2023-11-15T06:41:24.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 msgr2=0x7fa58c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:24.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c109450 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7fa580013510 tx=0x7fa58000b040).stop 2023-11-15T06:41:24.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 shutdown_connections 2023-11-15T06:41:24.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa57402ff60 0x7fa574032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:24.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 --2- 172.21.15.135:0/3304800992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa58c0f5410 0x7fa58c109450 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:24.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 >> 172.21.15.135:0/3304800992 conn(0x7fa58c0f0b70 msgr2=0x7fa58c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:24.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 shutdown_connections 2023-11-15T06:41:24.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:24.667+0000 7fa590d7d700 1 -- 172.21.15.135:0/3304800992 wait complete. 2023-11-15T06:41:24.672 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:24.265247+0000 mgr.smithi123.razjtn (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:24.666169+0000 mon.smithi123 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.135:0/3304800992' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:26.087 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:26.087 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:27.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.181+0000 7f9ed3e25700 1 -- 172.21.15.135:0/1626187541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 msgr2=0x7f9ecc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:27.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.181+0000 7f9ed3e25700 1 --2- 172.21.15.135:0/1626187541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc0f5830 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f9ebc009fb0 tx=0x7f9ebc009b30).stop 2023-11-15T06:41:27.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.181+0000 7f9ed3e25700 1 -- 172.21.15.135:0/1626187541 shutdown_connections 2023-11-15T06:41:27.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.181+0000 7f9ed3e25700 1 --2- 172.21.15.135:0/1626187541 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc0f5830 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:27.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.181+0000 7f9ed3e25700 1 -- 172.21.15.135:0/1626187541 >> 172.21.15.135:0/1626187541 conn(0x7f9ecc0f0b70 msgr2=0x7f9ecc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:27.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed3e25700 1 -- 172.21.15.135:0/1626187541 shutdown_connections 2023-11-15T06:41:27.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed3e25700 1 -- 172.21.15.135:0/1626187541 wait complete. 2023-11-15T06:41:27.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed3e25700 1 Processor -- start 2023-11-15T06:41:27.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed3e25700 1 -- start start 2023-11-15T06:41:27.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed3e25700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:27.188 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed3e25700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ecc060220 con 0x7f9ecc0f5410 2023-11-15T06:41:27.188 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed1bc1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:27.188 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed1bc1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:47330/0 (socket says 172.21.15.135:47330) 2023-11-15T06:41:27.188 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.182+0000 7f9ed1bc1700 1 -- 172.21.15.135:0/3339756400 learned_addr learned my addr 172.21.15.135:0/3339756400 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:27.188 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.183+0000 7f9ed1bc1700 1 -- 172.21.15.135:0/3339756400 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9ebc005040 con 0x7f9ecc0f5410 2023-11-15T06:41:27.190 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.183+0000 7f9ed1bc1700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc109420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9ebc013510 tx=0x7f9ebc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:27.190 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.183+0000 7f9ec27fc700 1 -- 172.21.15.135:0/3339756400 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc009320 con 0x7f9ecc0f5410 2023-11-15T06:41:27.190 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.183+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ecc060420 con 0x7f9ecc0f5410 2023-11-15T06:41:27.190 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.183+0000 7f9ec27fc700 1 -- 172.21.15.135:0/3339756400 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9ebc013f00 con 0x7f9ecc0f5410 2023-11-15T06:41:27.191 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.183+0000 7f9ec27fc700 1 -- 172.21.15.135:0/3339756400 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9ebc014d90 con 0x7f9ecc0f5410 2023-11-15T06:41:27.191 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.183+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ecc109ad0 con 0x7f9ecc0f5410 2023-11-15T06:41:27.191 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.184+0000 7f9ec27fc700 1 -- 172.21.15.135:0/3339756400 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9ebc0263a0 con 0x7f9ecc0f5410 2023-11-15T06:41:27.191 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.184+0000 7f9ec27fc700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9eb802ff60 0x7f9eb8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:27.191 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.184+0000 7f9ec27fc700 1 -- 172.21.15.135:0/3339756400 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9ebc0412e0 con 0x7f9ecc0f5410 2023-11-15T06:41:27.192 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.184+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ecc059100 con 0x7f9ecc0f5410 2023-11-15T06:41:27.192 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.184+0000 7f9ed13c0700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9eb802ff60 0x7f9eb8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:27.192 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.185+0000 7f9ed13c0700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9eb802ff60 0x7f9eb8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9ec80096a0 tx=0x7f9ec8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:27.193 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.189+0000 7f9ec27fc700 1 -- 172.21.15.135:0/3339756400 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9ebc012920 con 0x7f9ecc0f5410 2023-11-15T06:41:27.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.398+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ecc10a0e0 con 0x7f9ecc0f5410 2023-11-15T06:41:27.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.398+0000 7f9ec27fc700 1 -- 172.21.15.135:0/3339756400 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9ebc02b030 con 0x7f9ecc0f5410 2023-11-15T06:41:27.402 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:27.402 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:27.403 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9eb802ff60 msgr2=0x7f9eb8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:27.403 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9eb802ff60 0x7f9eb8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f9ec80096a0 tx=0x7f9ec8006b40).stop 2023-11-15T06:41:27.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 msgr2=0x7f9ecc109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:27.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc109420 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f9ebc013510 tx=0x7f9ebc00b040).stop 2023-11-15T06:41:27.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 shutdown_connections 2023-11-15T06:41:27.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9eb802ff60 0x7f9eb8032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:27.404 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 --2- 172.21.15.135:0/3339756400 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ecc0f5410 0x7f9ecc109420 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:27.405 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 >> 172.21.15.135:0/3339756400 conn(0x7f9ecc0f0b70 msgr2=0x7f9ecc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:27.405 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 shutdown_connections 2023-11-15T06:41:27.405 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:27.400+0000 7f9ed3e25700 1 -- 172.21.15.135:0/3339756400 wait complete. 2023-11-15T06:41:27.406 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:27.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:26.265552+0000 mgr.smithi123.razjtn (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:27.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:27.399513+0000 mon.smithi123 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.135:0/3339756400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:28.791 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:28.792 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:29.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:28.265855+0000 mgr.smithi123.razjtn (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:29.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.940+0000 7f5007fff700 1 -- 172.21.15.135:0/2487415469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 msgr2=0x7f50080f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:29.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.940+0000 7f5007fff700 1 --2- 172.21.15.135:0/2487415469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50080f66c0 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f4ff8009fb0 tx=0x7f4ff8009b30).stop 2023-11-15T06:41:29.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.940+0000 7f5007fff700 1 -- 172.21.15.135:0/2487415469 shutdown_connections 2023-11-15T06:41:29.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.940+0000 7f5007fff700 1 --2- 172.21.15.135:0/2487415469 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50080f66c0 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:29.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.940+0000 7f5007fff700 1 -- 172.21.15.135:0/2487415469 >> 172.21.15.135:0/2487415469 conn(0x7f50080f1aa0 msgr2=0x7f50080f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:29.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.940+0000 7f5007fff700 1 -- 172.21.15.135:0/2487415469 shutdown_connections 2023-11-15T06:41:29.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.940+0000 7f5007fff700 1 -- 172.21.15.135:0/2487415469 wait complete. 2023-11-15T06:41:29.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.941+0000 7f5007fff700 1 Processor -- start 2023-11-15T06:41:29.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.941+0000 7f5007fff700 1 -- start start 2023-11-15T06:41:29.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.941+0000 7f5007fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50081064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:29.948 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.941+0000 7f5007fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50081069a0 con 0x7f50080f62a0 2023-11-15T06:41:29.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.941+0000 7f5006ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50081064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:29.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.941+0000 7f5006ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50081064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43970/0 (socket says 172.21.15.135:43970) 2023-11-15T06:41:29.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.941+0000 7f5006ffd700 1 -- 172.21.15.135:0/661811885 learned_addr learned my addr 172.21.15.135:0/661811885 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:29.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.942+0000 7f5006ffd700 1 -- 172.21.15.135:0/661811885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff8005040 con 0x7f50080f62a0 2023-11-15T06:41:29.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.942+0000 7f5006ffd700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50081064b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f4ff8013510 tx=0x7f4ff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:29.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.942+0000 7f4fef7fe700 1 -- 172.21.15.135:0/661811885 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff8009320 con 0x7f50080f62a0 2023-11-15T06:41:29.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.942+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5008106ba0 con 0x7f50080f62a0 2023-11-15T06:41:29.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.942+0000 7f4fef7fe700 1 -- 172.21.15.135:0/661811885 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ff8013f00 con 0x7f50080f62a0 2023-11-15T06:41:29.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.942+0000 7f4fef7fe700 1 -- 172.21.15.135:0/661811885 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff8014dc0 con 0x7f50080f62a0 2023-11-15T06:41:29.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.942+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5008107030 con 0x7f50080f62a0 2023-11-15T06:41:29.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.943+0000 7f4fef7fe700 1 -- 172.21.15.135:0/661811885 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ff801d3d0 con 0x7f50080f62a0 2023-11-15T06:41:29.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.943+0000 7f4fef7fe700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4ff002ff60 0x7f4ff0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:29.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.943+0000 7f4fef7fe700 1 -- 172.21.15.135:0/661811885 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ff80410f0 con 0x7f50080f62a0 2023-11-15T06:41:29.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.943+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5008045ad0 con 0x7f50080f62a0 2023-11-15T06:41:29.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.943+0000 7f50067fc700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4ff002ff60 0x7f4ff0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:29.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.944+0000 7f50067fc700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4ff002ff60 0x7f4ff0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f4ffc0096a0 tx=0x7f4ffc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:29.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:29.948+0000 7f4fef7fe700 1 -- 172.21.15.135:0/661811885 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ff8012920 con 0x7f50080f62a0 2023-11-15T06:41:30.159 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.156+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50080fbdd0 con 0x7f50080f62a0 2023-11-15T06:41:30.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.157+0000 7f4fef7fe700 1 -- 172.21.15.135:0/661811885 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4ff802b020 con 0x7f50080f62a0 2023-11-15T06:41:30.163 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:30.163 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:30.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4ff002ff60 msgr2=0x7f4ff0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:30.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4ff002ff60 0x7f4ff0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f4ffc0096a0 tx=0x7f4ffc006b40).stop 2023-11-15T06:41:30.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 msgr2=0x7f50081064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:30.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50081064b0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f4ff8013510 tx=0x7f4ff800b040).stop 2023-11-15T06:41:30.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 shutdown_connections 2023-11-15T06:41:30.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4ff002ff60 0x7f4ff0032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:30.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 --2- 172.21.15.135:0/661811885 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50080f62a0 0x7f50081064b0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:30.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 >> 172.21.15.135:0/661811885 conn(0x7f50080f1aa0 msgr2=0x7f50080f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:30.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 shutdown_connections 2023-11-15T06:41:30.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:30.159+0000 7f5007fff700 1 -- 172.21.15.135:0/661811885 wait complete. 2023-11-15T06:41:30.166 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:30.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:30 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:30.158019+0000 mon.smithi123 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.135:0/661811885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:31.533 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:31.534 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:31.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:30.266044+0000 mgr.smithi123.razjtn (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:32.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.709+0000 7f7705fbf700 1 -- 172.21.15.135:0/593687368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 msgr2=0x7f77000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:32.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.709+0000 7f7705fbf700 1 --2- 172.21.15.135:0/593687368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77000f5830 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f76f4009fb0 tx=0x7f76f4009b30).stop 2023-11-15T06:41:32.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.709+0000 7f7705fbf700 1 -- 172.21.15.135:0/593687368 shutdown_connections 2023-11-15T06:41:32.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.709+0000 7f7705fbf700 1 --2- 172.21.15.135:0/593687368 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77000f5830 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:32.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.709+0000 7f7705fbf700 1 -- 172.21.15.135:0/593687368 >> 172.21.15.135:0/593687368 conn(0x7f77000f0b70 msgr2=0x7f77000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:32.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.709+0000 7f7705fbf700 1 -- 172.21.15.135:0/593687368 shutdown_connections 2023-11-15T06:41:32.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.709+0000 7f7705fbf700 1 -- 172.21.15.135:0/593687368 wait complete. 2023-11-15T06:41:32.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.710+0000 7f7705fbf700 1 Processor -- start 2023-11-15T06:41:32.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.710+0000 7f7705fbf700 1 -- start start 2023-11-15T06:41:32.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.710+0000 7f7705fbf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77001093d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:32.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.710+0000 7f7705fbf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7700060240 con 0x7f77000f5410 2023-11-15T06:41:32.718 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.710+0000 7f76ff7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77001093d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:32.718 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f76ff7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77001093d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:43988/0 (socket says 172.21.15.135:43988) 2023-11-15T06:41:32.718 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f76ff7fe700 1 -- 172.21.15.135:0/3059537224 learned_addr learned my addr 172.21.15.135:0/3059537224 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:32.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f76ff7fe700 1 -- 172.21.15.135:0/3059537224 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f4005040 con 0x7f77000f5410 2023-11-15T06:41:32.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f76ff7fe700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77001093d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f76f4013510 tx=0x7f76f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:32.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f76e7fff700 1 -- 172.21.15.135:0/3059537224 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f4009320 con 0x7f77000f5410 2023-11-15T06:41:32.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f76e7fff700 1 -- 172.21.15.135:0/3059537224 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76f4013f00 con 0x7f77000f5410 2023-11-15T06:41:32.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7700060440 con 0x7f77000f5410 2023-11-15T06:41:32.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f76e7fff700 1 -- 172.21.15.135:0/3059537224 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f4014d90 con 0x7f77000f5410 2023-11-15T06:41:32.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.711+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7700109a80 con 0x7f77000f5410 2023-11-15T06:41:32.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.712+0000 7f76e7fff700 1 -- 172.21.15.135:0/3059537224 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76f40095f0 con 0x7f77000f5410 2023-11-15T06:41:32.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.712+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7700045ad0 con 0x7f77000f5410 2023-11-15T06:41:32.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.712+0000 7f76e7fff700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e802ff60 0x7f76e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:32.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.712+0000 7f76e7fff700 1 -- 172.21.15.135:0/3059537224 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76f4040fe0 con 0x7f77000f5410 2023-11-15T06:41:32.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.712+0000 7f76feffd700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e802ff60 0x7f76e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:32.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.713+0000 7f76feffd700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e802ff60 0x7f76e8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f76f00096a0 tx=0x7f76f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:32.728 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.717+0000 7f76e7fff700 1 -- 172.21.15.135:0/3059537224 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76f401d7f0 con 0x7f77000f5410 2023-11-15T06:41:32.926 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.923+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f770010a2a0 con 0x7f77000f5410 2023-11-15T06:41:32.927 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.924+0000 7f76e7fff700 1 -- 172.21.15.135:0/3059537224 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76f4010230 con 0x7f77000f5410 2023-11-15T06:41:32.927 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:32.927 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:32.928 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e802ff60 msgr2=0x7f76e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:32.929 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e802ff60 0x7f76e8032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f76f00096a0 tx=0x7f76f0006b40).stop 2023-11-15T06:41:32.929 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 msgr2=0x7f77001093d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:32.929 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77001093d0 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f76f4013510 tx=0x7f76f400b040).stop 2023-11-15T06:41:32.929 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 shutdown_connections 2023-11-15T06:41:32.929 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e802ff60 0x7f76e8032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:32.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 --2- 172.21.15.135:0/3059537224 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f77000f5410 0x7f77001093d0 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:32.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 >> 172.21.15.135:0/3059537224 conn(0x7f77000f0b70 msgr2=0x7f77000f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:32.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 shutdown_connections 2023-11-15T06:41:32.930 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:32.926+0000 7f7705fbf700 1 -- 172.21.15.135:0/3059537224 wait complete. 2023-11-15T06:41:32.931 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:33.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:32.266361+0000 mgr.smithi123.razjtn (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:33.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:32.925286+0000 mon.smithi123 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.135:0/3059537224' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:34.303 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:34.304 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:35.422 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 -- 172.21.15.135:0/2719133771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 msgr2=0x7fd8a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:35.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 --2- 172.21.15.135:0/2719133771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a40f55c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7fd894009fb0 tx=0x7fd894009b30).stop 2023-11-15T06:41:35.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 -- 172.21.15.135:0/2719133771 shutdown_connections 2023-11-15T06:41:35.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 --2- 172.21.15.135:0/2719133771 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a40f55c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:35.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 -- 172.21.15.135:0/2719133771 >> 172.21.15.135:0/2719133771 conn(0x7fd8a40f09a0 msgr2=0x7fd8a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:35.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 -- 172.21.15.135:0/2719133771 shutdown_connections 2023-11-15T06:41:35.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 -- 172.21.15.135:0/2719133771 wait complete. 2023-11-15T06:41:35.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 Processor -- start 2023-11-15T06:41:35.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 -- start start 2023-11-15T06:41:35.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a41091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:35.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.419+0000 7fd8a359e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8a4060240 con 0x7fd8a40f51a0 2023-11-15T06:41:35.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd8a259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a41091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:35.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd8a259c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a41091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44006/0 (socket says 172.21.15.135:44006) 2023-11-15T06:41:35.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd8a259c700 1 -- 172.21.15.135:0/3996541665 learned_addr learned my addr 172.21.15.135:0/3996541665 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:35.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd8a259c700 1 -- 172.21.15.135:0/3996541665 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd894005040 con 0x7fd8a40f51a0 2023-11-15T06:41:35.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd8a259c700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a41091b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd894013510 tx=0x7fd89400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:35.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd89affd700 1 -- 172.21.15.135:0/3996541665 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd894009320 con 0x7fd8a40f51a0 2023-11-15T06:41:35.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8a4060440 con 0x7fd8a40f51a0 2023-11-15T06:41:35.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd89affd700 1 -- 172.21.15.135:0/3996541665 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd894013f00 con 0x7fd8a40f51a0 2023-11-15T06:41:35.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd89affd700 1 -- 172.21.15.135:0/3996541665 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd894014dc0 con 0x7fd8a40f51a0 2023-11-15T06:41:35.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.420+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8a4109860 con 0x7fd8a40f51a0 2023-11-15T06:41:35.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.421+0000 7fd89affd700 1 -- 172.21.15.135:0/3996541665 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8940095f0 con 0x7fd8a40f51a0 2023-11-15T06:41:35.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.421+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8a4045ad0 con 0x7fd8a40f51a0 2023-11-15T06:41:35.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.421+0000 7fd89affd700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd89002ff60 0x7fd890032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:35.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.421+0000 7fd89affd700 1 -- 172.21.15.135:0/3996541665 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd894041000 con 0x7fd8a40f51a0 2023-11-15T06:41:35.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.421+0000 7fd8a1d9b700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd89002ff60 0x7fd890032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:35.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.422+0000 7fd8a1d9b700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd89002ff60 0x7fd890032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd88c0096a0 tx=0x7fd88c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:35.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.426+0000 7fd89affd700 1 -- 172.21.15.135:0/3996541665 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd89401c790 con 0x7fd8a40f51a0 2023-11-15T06:41:35.633 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.631+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8a410a080 con 0x7fd8a40f51a0 2023-11-15T06:41:35.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.631+0000 7fd89affd700 1 -- 172.21.15.135:0/3996541665 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd894010230 con 0x7fd8a40f51a0 2023-11-15T06:41:35.634 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:35.635 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:35.635 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd89002ff60 msgr2=0x7fd890032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:35.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd89002ff60 0x7fd890032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fd88c0096a0 tx=0x7fd88c006b40).stop 2023-11-15T06:41:35.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 msgr2=0x7fd8a41091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:35.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a41091b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7fd894013510 tx=0x7fd89400b040).stop 2023-11-15T06:41:35.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 shutdown_connections 2023-11-15T06:41:35.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd89002ff60 0x7fd890032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:35.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 --2- 172.21.15.135:0/3996541665 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd8a40f51a0 0x7fd8a41091b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:35.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 >> 172.21.15.135:0/3996541665 conn(0x7fd8a40f09a0 msgr2=0x7fd8a40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:35.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 shutdown_connections 2023-11-15T06:41:35.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:35.633+0000 7fd8a359e700 1 -- 172.21.15.135:0/3996541665 wait complete. 2023-11-15T06:41:35.638 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:35.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:34.266760+0000 mgr.smithi123.razjtn (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:36.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:36 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:35.632451+0000 mon.smithi123 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.135:0/3996541665' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:37.028 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:37.029 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:37.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:36.267113+0000 mgr.smithi123.razjtn (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:38.218 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.215+0000 7fee90b9c700 1 -- 172.21.15.135:0/106229299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 msgr2=0x7fee8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:38.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.215+0000 7fee90b9c700 1 --2- 172.21.15.135:0/106229299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c0f5830 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7fee74009fb0 tx=0x7fee74009b30).stop 2023-11-15T06:41:38.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.215+0000 7fee90b9c700 1 -- 172.21.15.135:0/106229299 shutdown_connections 2023-11-15T06:41:38.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.215+0000 7fee90b9c700 1 --2- 172.21.15.135:0/106229299 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c0f5830 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:38.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.215+0000 7fee90b9c700 1 -- 172.21.15.135:0/106229299 >> 172.21.15.135:0/106229299 conn(0x7fee8c0f0b70 msgr2=0x7fee8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:38.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.216+0000 7fee90b9c700 1 -- 172.21.15.135:0/106229299 shutdown_connections 2023-11-15T06:41:38.220 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.216+0000 7fee90b9c700 1 -- 172.21.15.135:0/106229299 wait complete. 2023-11-15T06:41:38.221 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.216+0000 7fee90b9c700 1 Processor -- start 2023-11-15T06:41:38.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.216+0000 7fee90b9c700 1 -- start start 2023-11-15T06:41:38.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.216+0000 7fee90b9c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:38.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.216+0000 7fee90b9c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee8c106c20 con 0x7fee8c0f5410 2023-11-15T06:41:38.222 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee8a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:38.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee8a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44016/0 (socket says 172.21.15.135:44016) 2023-11-15T06:41:38.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee8a59c700 1 -- 172.21.15.135:0/4073775680 learned_addr learned my addr 172.21.15.135:0/4073775680 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:38.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee8a59c700 1 -- 172.21.15.135:0/4073775680 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee74005040 con 0x7fee8c0f5410 2023-11-15T06:41:38.223 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee8a59c700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fee74013510 tx=0x7fee7400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:38.224 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee82ffd700 1 -- 172.21.15.135:0/4073775680 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee74009320 con 0x7fee8c0f5410 2023-11-15T06:41:38.224 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee82ffd700 1 -- 172.21.15.135:0/4073775680 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee74013f00 con 0x7fee8c0f5410 2023-11-15T06:41:38.224 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee8c106e20 con 0x7fee8c0f5410 2023-11-15T06:41:38.224 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee82ffd700 1 -- 172.21.15.135:0/4073775680 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee74014d90 con 0x7fee8c0f5410 2023-11-15T06:41:38.224 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.217+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee8c1072b0 con 0x7fee8c0f5410 2023-11-15T06:41:38.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.218+0000 7fee82ffd700 1 -- 172.21.15.135:0/4073775680 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee74014420 con 0x7fee8c0f5410 2023-11-15T06:41:38.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.218+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee8c045ad0 con 0x7fee8c0f5410 2023-11-15T06:41:38.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.218+0000 7fee82ffd700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee7802ff60 0x7fee78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:38.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.218+0000 7fee82ffd700 1 -- 172.21.15.135:0/4073775680 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee740410c0 con 0x7fee8c0f5410 2023-11-15T06:41:38.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.219+0000 7fee89d9b700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee7802ff60 0x7fee78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:38.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.219+0000 7fee89d9b700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee7802ff60 0x7fee78032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fee7c0096a0 tx=0x7fee7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:38.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.223+0000 7fee82ffd700 1 -- 172.21.15.135:0/4073775680 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee740095f0 con 0x7fee8c0f5410 2023-11-15T06:41:38.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.429+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee8c0fbf90 con 0x7fee8c0f5410 2023-11-15T06:41:38.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.430+0000 7fee82ffd700 1 -- 172.21.15.135:0/4073775680 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fee7401db70 con 0x7fee8c0f5410 2023-11-15T06:41:38.433 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:38.433 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:38.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee7802ff60 msgr2=0x7fee78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:38.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee7802ff60 0x7fee78032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fee7c0096a0 tx=0x7fee7c006b40).stop 2023-11-15T06:41:38.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 msgr2=0x7fee8c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:38.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7fee74013510 tx=0x7fee7400b040).stop 2023-11-15T06:41:38.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 shutdown_connections 2023-11-15T06:41:38.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee7802ff60 0x7fee78032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:38.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 --2- 172.21.15.135:0/4073775680 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee8c0f5410 0x7fee8c106730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:38.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.431+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 >> 172.21.15.135:0/4073775680 conn(0x7fee8c0f0b70 msgr2=0x7fee8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:38.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.432+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 shutdown_connections 2023-11-15T06:41:38.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:38.432+0000 7fee90b9c700 1 -- 172.21.15.135:0/4073775680 wait complete. 2023-11-15T06:41:38.437 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:38.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:38 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:38.430812+0000 mon.smithi123 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.135:0/4073775680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:39.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:39.807 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:39.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:38.267477+0000 mgr.smithi123.razjtn (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:40.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:40.365866+0000 mon.smithi123 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:41:40.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:40.372250+0000 mon.smithi123 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:41:40.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.952+0000 7f7a41be1700 1 -- 172.21.15.135:0/276485683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 msgr2=0x7f7a3c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:40.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.952+0000 7f7a41be1700 1 --2- 172.21.15.135:0/276485683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c0f90f0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7f7a30009fb0 tx=0x7f7a30009b30).stop 2023-11-15T06:41:40.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.952+0000 7f7a41be1700 1 -- 172.21.15.135:0/276485683 shutdown_connections 2023-11-15T06:41:40.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.952+0000 7f7a41be1700 1 --2- 172.21.15.135:0/276485683 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c0f90f0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:40.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.952+0000 7f7a41be1700 1 -- 172.21.15.135:0/276485683 >> 172.21.15.135:0/276485683 conn(0x7f7a3c0f0b70 msgr2=0x7f7a3c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:40.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.952+0000 7f7a41be1700 1 -- 172.21.15.135:0/276485683 shutdown_connections 2023-11-15T06:41:40.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.952+0000 7f7a41be1700 1 -- 172.21.15.135:0/276485683 wait complete. 2023-11-15T06:41:40.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.953+0000 7f7a41be1700 1 Processor -- start 2023-11-15T06:41:40.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.953+0000 7f7a41be1700 1 -- start start 2023-11-15T06:41:40.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.953+0000 7f7a41be1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:40.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.953+0000 7f7a41be1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a3c106bd0 con 0x7f7a3c060280 2023-11-15T06:41:40.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.953+0000 7f7a3b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:40.959 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.953+0000 7f7a3b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48674/0 (socket says 172.21.15.135:48674) 2023-11-15T06:41:40.959 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.953+0000 7f7a3b7fe700 1 -- 172.21.15.135:0/15149018 learned_addr learned my addr 172.21.15.135:0/15149018 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:40.959 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.954+0000 7f7a3b7fe700 1 -- 172.21.15.135:0/15149018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a30005040 con 0x7f7a3c060280 2023-11-15T06:41:40.960 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.954+0000 7f7a3b7fe700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c1066e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7a30009830 tx=0x7f7a3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:40.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.954+0000 7f7a23fff700 1 -- 172.21.15.135:0/15149018 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a30009320 con 0x7f7a3c060280 2023-11-15T06:41:40.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.954+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a3c106dd0 con 0x7f7a3c060280 2023-11-15T06:41:40.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.954+0000 7f7a23fff700 1 -- 172.21.15.135:0/15149018 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a300136c0 con 0x7f7a3c060280 2023-11-15T06:41:40.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.954+0000 7f7a23fff700 1 -- 172.21.15.135:0/15149018 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a3001ee90 con 0x7f7a3c060280 2023-11-15T06:41:40.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.954+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a3c107260 con 0x7f7a3c060280 2023-11-15T06:41:40.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.955+0000 7f7a23fff700 1 -- 172.21.15.135:0/15149018 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a30014420 con 0x7f7a3c060280 2023-11-15T06:41:40.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.955+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a3c045ad0 con 0x7f7a3c060280 2023-11-15T06:41:40.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.955+0000 7f7a23fff700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7a2402ff10 0x7f7a240323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:40.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.955+0000 7f7a23fff700 1 -- 172.21.15.135:0/15149018 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a300412e0 con 0x7f7a3c060280 2023-11-15T06:41:40.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.955+0000 7f7a3affd700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7a2402ff10 0x7f7a240323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:40.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.955+0000 7f7a3affd700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7a2402ff10 0x7f7a240323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7a2c0096a0 tx=0x7f7a2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:40.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:40.960+0000 7f7a23fff700 1 -- 172.21.15.135:0/15149018 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a3001c6a0 con 0x7f7a3c060280 2023-11-15T06:41:41.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.166+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7a3c108110 con 0x7f7a3c060280 2023-11-15T06:41:41.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.167+0000 7f7a23fff700 1 -- 172.21.15.135:0/15149018 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7a3001c6a0 con 0x7f7a3c060280 2023-11-15T06:41:41.170 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:41.170 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:41.171 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7a2402ff10 msgr2=0x7f7a240323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:41.171 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7a2402ff10 0x7f7a240323d0 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7f7a2c0096a0 tx=0x7f7a2c006b40).stop 2023-11-15T06:41:41.171 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 msgr2=0x7f7a3c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:41.172 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c1066e0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7f7a30009830 tx=0x7f7a3000b040).stop 2023-11-15T06:41:41.172 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 shutdown_connections 2023-11-15T06:41:41.172 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7a2402ff10 0x7f7a240323d0 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:41.172 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 --2- 172.21.15.135:0/15149018 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7a3c060280 0x7f7a3c1066e0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:41.172 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 >> 172.21.15.135:0/15149018 conn(0x7f7a3c0f0b70 msgr2=0x7f7a3c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:41.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 shutdown_connections 2023-11-15T06:41:41.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:41.168+0000 7f7a41be1700 1 -- 172.21.15.135:0/15149018 wait complete. 2023-11-15T06:41:41.173 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:41.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:40.267804+0000 mgr.smithi123.razjtn (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:41.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:41.167834+0000 mon.smithi123 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.135:0/15149018' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:42.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:42.542 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:43.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.687+0000 7fb561bfe700 1 -- 172.21.15.135:0/3218031737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 msgr2=0x7fb55c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:43.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.687+0000 7fb561bfe700 1 --2- 172.21.15.135:0/3218031737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c0f5830 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7fb550009fb0 tx=0x7fb550009b30).stop 2023-11-15T06:41:43.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.687+0000 7fb561bfe700 1 -- 172.21.15.135:0/3218031737 shutdown_connections 2023-11-15T06:41:43.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.687+0000 7fb561bfe700 1 --2- 172.21.15.135:0/3218031737 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c0f5830 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:43.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.687+0000 7fb561bfe700 1 -- 172.21.15.135:0/3218031737 >> 172.21.15.135:0/3218031737 conn(0x7fb55c0f0b70 msgr2=0x7fb55c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:43.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.687+0000 7fb561bfe700 1 -- 172.21.15.135:0/3218031737 shutdown_connections 2023-11-15T06:41:43.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.687+0000 7fb561bfe700 1 -- 172.21.15.135:0/3218031737 wait complete. 2023-11-15T06:41:43.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb561bfe700 1 Processor -- start 2023-11-15T06:41:43.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb561bfe700 1 -- start start 2023-11-15T06:41:43.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb561bfe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:43.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb561bfe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb55c105b20 con 0x7fb55c0f5410 2023-11-15T06:41:43.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb55b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:43.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb55b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48686/0 (socket says 172.21.15.135:48686) 2023-11-15T06:41:43.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb55b7fe700 1 -- 172.21.15.135:0/2486934008 learned_addr learned my addr 172.21.15.135:0/2486934008 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:43.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.688+0000 7fb55b7fe700 1 -- 172.21.15.135:0/2486934008 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb550005040 con 0x7fb55c0f5410 2023-11-15T06:41:43.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.689+0000 7fb55b7fe700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c105630 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb550013510 tx=0x7fb55000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:43.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.689+0000 7fb543fff700 1 -- 172.21.15.135:0/2486934008 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb550009320 con 0x7fb55c0f5410 2023-11-15T06:41:43.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.689+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb55c105d20 con 0x7fb55c0f5410 2023-11-15T06:41:43.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.689+0000 7fb543fff700 1 -- 172.21.15.135:0/2486934008 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb550013f00 con 0x7fb55c0f5410 2023-11-15T06:41:43.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.689+0000 7fb543fff700 1 -- 172.21.15.135:0/2486934008 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb550014d90 con 0x7fb55c0f5410 2023-11-15T06:41:43.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.689+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb55c1061b0 con 0x7fb55c0f5410 2023-11-15T06:41:43.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.690+0000 7fb543fff700 1 -- 172.21.15.135:0/2486934008 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb5500095f0 con 0x7fb55c0f5410 2023-11-15T06:41:43.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.690+0000 7fb543fff700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb54402ff60 0x7fb544032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:43.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.690+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb55c045ad0 con 0x7fb55c0f5410 2023-11-15T06:41:43.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.690+0000 7fb543fff700 1 -- 172.21.15.135:0/2486934008 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb550040fc0 con 0x7fb55c0f5410 2023-11-15T06:41:43.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.690+0000 7fb55affd700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb54402ff60 0x7fb544032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:43.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.690+0000 7fb55affd700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb54402ff60 0x7fb544032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:43.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.694+0000 7fb543fff700 1 -- 172.21.15.135:0/2486934008 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb550014420 con 0x7fb55c0f5410 2023-11-15T06:41:43.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.899+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb55c0faf50 con 0x7fb55c0f5410 2023-11-15T06:41:43.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.899+0000 7fb543fff700 1 -- 172.21.15.135:0/2486934008 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb550010230 con 0x7fb55c0f5410 2023-11-15T06:41:43.903 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:43.903 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:43.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb54402ff60 msgr2=0x7fb544032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:43.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb54402ff60 0x7fb544032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7fb54c0096a0 tx=0x7fb54c006b40).stop 2023-11-15T06:41:43.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 msgr2=0x7fb55c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:43.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c105630 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7fb550013510 tx=0x7fb55000b040).stop 2023-11-15T06:41:43.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 shutdown_connections 2023-11-15T06:41:43.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb54402ff60 0x7fb544032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:43.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 --2- 172.21.15.135:0/2486934008 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb55c0f5410 0x7fb55c105630 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:43.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 >> 172.21.15.135:0/2486934008 conn(0x7fb55c0f0b70 msgr2=0x7fb55c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:43.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 shutdown_connections 2023-11-15T06:41:43.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:43.901+0000 7fb561bfe700 1 -- 172.21.15.135:0/2486934008 wait complete. 2023-11-15T06:41:43.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:42.268194+0000 mgr.smithi123.razjtn (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:43.906 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:44.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:44 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:43.900526+0000 mon.smithi123 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.135:0/2486934008' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:45.282 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:45.283 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:45.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:44.268560+0000 mgr.smithi123.razjtn (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:46.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 -- 172.21.15.135:0/2362657237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 msgr2=0x7f45600f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:46.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 --2- 172.21.15.135:0/2362657237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45600f90f0 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f4550009fb0 tx=0x7f4550009b30).stop 2023-11-15T06:41:46.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 -- 172.21.15.135:0/2362657237 shutdown_connections 2023-11-15T06:41:46.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 --2- 172.21.15.135:0/2362657237 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45600f90f0 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:46.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 -- 172.21.15.135:0/2362657237 >> 172.21.15.135:0/2362657237 conn(0x7f45600f0b70 msgr2=0x7f45600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:46.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 -- 172.21.15.135:0/2362657237 shutdown_connections 2023-11-15T06:41:46.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 -- 172.21.15.135:0/2362657237 wait complete. 2023-11-15T06:41:46.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.431+0000 7f456627c700 1 Processor -- start 2023-11-15T06:41:46.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f456627c700 1 -- start start 2023-11-15T06:41:46.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f456627c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45601066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:46.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f456627c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4560106bd0 con 0x7f4560060280 2023-11-15T06:41:46.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f455ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45601066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:46.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f455ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45601066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48706/0 (socket says 172.21.15.135:48706) 2023-11-15T06:41:46.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f455ffff700 1 -- 172.21.15.135:0/571617311 learned_addr learned my addr 172.21.15.135:0/571617311 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:46.438 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f455ffff700 1 -- 172.21.15.135:0/571617311 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4550005040 con 0x7f4560060280 2023-11-15T06:41:46.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.432+0000 7f455ffff700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45601066e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f4550013510 tx=0x7f455000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:46.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.433+0000 7f455cff9700 1 -- 172.21.15.135:0/571617311 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4550009320 con 0x7f4560060280 2023-11-15T06:41:46.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.433+0000 7f455cff9700 1 -- 172.21.15.135:0/571617311 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4550013f00 con 0x7f4560060280 2023-11-15T06:41:46.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.433+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4560106dd0 con 0x7f4560060280 2023-11-15T06:41:46.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.433+0000 7f455cff9700 1 -- 172.21.15.135:0/571617311 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4550014dc0 con 0x7f4560060280 2023-11-15T06:41:46.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.433+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4560107260 con 0x7f4560060280 2023-11-15T06:41:46.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.433+0000 7f455cff9700 1 -- 172.21.15.135:0/571617311 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4550014420 con 0x7f4560060280 2023-11-15T06:41:46.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.433+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4560045ad0 con 0x7f4560060280 2023-11-15T06:41:46.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.434+0000 7f455cff9700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f454802ff60 0x7f4548032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:46.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.434+0000 7f455cff9700 1 -- 172.21.15.135:0/571617311 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4550041050 con 0x7f4560060280 2023-11-15T06:41:46.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.434+0000 7f455f7fe700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f454802ff60 0x7f4548032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:46.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.434+0000 7f455f7fe700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f454802ff60 0x7f4548032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f45540096a0 tx=0x7f4554006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:46.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.438+0000 7f455cff9700 1 -- 172.21.15.135:0/571617311 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45500095f0 con 0x7f4560060280 2023-11-15T06:41:46.649 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.646+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4560108110 con 0x7f4560060280 2023-11-15T06:41:46.650 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.647+0000 7f455cff9700 1 -- 172.21.15.135:0/571617311 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f455001cb10 con 0x7f4560060280 2023-11-15T06:41:46.650 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:46.650 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:46.651 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f454802ff60 msgr2=0x7f4548032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:46.651 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f454802ff60 0x7f4548032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f45540096a0 tx=0x7f4554006b40).stop 2023-11-15T06:41:46.651 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 msgr2=0x7f45601066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:46.652 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45601066e0 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f4550013510 tx=0x7f455000b040).stop 2023-11-15T06:41:46.652 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 shutdown_connections 2023-11-15T06:41:46.652 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f454802ff60 0x7f4548032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:46.652 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 --2- 172.21.15.135:0/571617311 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4560060280 0x7f45601066e0 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:46.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 >> 172.21.15.135:0/571617311 conn(0x7f45600f0b70 msgr2=0x7f45600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:46.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 shutdown_connections 2023-11-15T06:41:46.653 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:46.648+0000 7f456627c700 1 -- 172.21.15.135:0/571617311 wait complete. 2023-11-15T06:41:46.653 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:47.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:46.268913+0000 mgr.smithi123.razjtn (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:47.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:46.647968+0000 mon.smithi123 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.135:0/571617311' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:48.066 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:48.067 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:49.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.170+0000 7f0138914700 1 -- 172.21.15.135:0/1876560421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 msgr2=0x7f01340f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:49.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.170+0000 7f0138914700 1 --2- 172.21.15.135:0/1876560421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01340f66c0 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7f0124009fb0 tx=0x7f0124009b30).stop 2023-11-15T06:41:49.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.170+0000 7f0138914700 1 -- 172.21.15.135:0/1876560421 shutdown_connections 2023-11-15T06:41:49.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.170+0000 7f0138914700 1 --2- 172.21.15.135:0/1876560421 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01340f66c0 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:49.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.170+0000 7f0138914700 1 -- 172.21.15.135:0/1876560421 >> 172.21.15.135:0/1876560421 conn(0x7f01340f1ac0 msgr2=0x7f01340f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:49.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.170+0000 7f0138914700 1 -- 172.21.15.135:0/1876560421 shutdown_connections 2023-11-15T06:41:49.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.170+0000 7f0138914700 1 -- 172.21.15.135:0/1876560421 wait complete. 2023-11-15T06:41:49.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0138914700 1 Processor -- start 2023-11-15T06:41:49.177 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0138914700 1 -- start start 2023-11-15T06:41:49.177 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0138914700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01341064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:49.177 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0138914700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0134106990 con 0x7f01340f62a0 2023-11-15T06:41:49.177 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0132d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01341064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:49.177 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0132d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01341064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56754/0 (socket says 172.21.15.135:56754) 2023-11-15T06:41:49.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0132d9d700 1 -- 172.21.15.135:0/2660549592 learned_addr learned my addr 172.21.15.135:0/2660549592 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:49.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.171+0000 7f0132d9d700 1 -- 172.21.15.135:0/2660549592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0124005040 con 0x7f01340f62a0 2023-11-15T06:41:49.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.172+0000 7f0132d9d700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01341064a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0124013510 tx=0x7f012400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:49.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.172+0000 7f011b7fe700 1 -- 172.21.15.135:0/2660549592 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0124009320 con 0x7f01340f62a0 2023-11-15T06:41:49.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.172+0000 7f011b7fe700 1 -- 172.21.15.135:0/2660549592 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0124013f00 con 0x7f01340f62a0 2023-11-15T06:41:49.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.172+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0134106b90 con 0x7f01340f62a0 2023-11-15T06:41:49.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.172+0000 7f011b7fe700 1 -- 172.21.15.135:0/2660549592 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0124014dc0 con 0x7f01340f62a0 2023-11-15T06:41:49.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.172+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0134107020 con 0x7f01340f62a0 2023-11-15T06:41:49.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.173+0000 7f011b7fe700 1 -- 172.21.15.135:0/2660549592 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f01240095f0 con 0x7f01340f62a0 2023-11-15T06:41:49.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.173+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0134045ad0 con 0x7f01340f62a0 2023-11-15T06:41:49.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.173+0000 7f011b7fe700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f011c02ff60 0x7f011c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:49.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.173+0000 7f011b7fe700 1 -- 172.21.15.135:0/2660549592 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0124041040 con 0x7f01340f62a0 2023-11-15T06:41:49.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.173+0000 7f013259c700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f011c02ff60 0x7f011c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:49.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.173+0000 7f013259c700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f011c02ff60 0x7f011c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f01280096a0 tx=0x7f0128006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:49.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.177+0000 7f011b7fe700 1 -- 172.21.15.135:0/2660549592 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0124014420 con 0x7f01340f62a0 2023-11-15T06:41:49.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.383+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0134107ed0 con 0x7f01340f62a0 2023-11-15T06:41:49.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.384+0000 7f011b7fe700 1 -- 172.21.15.135:0/2660549592 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0124010230 con 0x7f01340f62a0 2023-11-15T06:41:49.387 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:49.387 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:49.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f011c02ff60 msgr2=0x7f011c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:49.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f011c02ff60 0x7f011c032420 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f01280096a0 tx=0x7f0128006b40).stop 2023-11-15T06:41:49.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 msgr2=0x7f01341064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:49.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01341064a0 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7f0124013510 tx=0x7f012400b040).stop 2023-11-15T06:41:49.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 shutdown_connections 2023-11-15T06:41:49.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f011c02ff60 0x7f011c032420 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:49.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 --2- 172.21.15.135:0/2660549592 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f01340f62a0 0x7f01341064a0 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:49.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 >> 172.21.15.135:0/2660549592 conn(0x7f01340f1ac0 msgr2=0x7f01340f2560 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:49.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 shutdown_connections 2023-11-15T06:41:49.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:49.386+0000 7f0138914700 1 -- 172.21.15.135:0/2660549592 wait complete. 2023-11-15T06:41:49.391 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:48.269293+0000 mgr.smithi123.razjtn (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:49.385144+0000 mon.smithi123 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.135:0/2660549592' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:50.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:50.787 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:51.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:50.269645+0000 mgr.smithi123.razjtn (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:51.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.947+0000 7f12b89c3700 1 -- 172.21.15.135:0/3538536437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 msgr2=0x7f12b40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:51.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.947+0000 7f12b89c3700 1 --2- 172.21.15.135:0/3538536437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b40f55c0 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f12a4009fb0 tx=0x7f12a4009b30).stop 2023-11-15T06:41:51.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.947+0000 7f12b89c3700 1 -- 172.21.15.135:0/3538536437 shutdown_connections 2023-11-15T06:41:51.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.947+0000 7f12b89c3700 1 --2- 172.21.15.135:0/3538536437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b40f55c0 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:51.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.948+0000 7f12b89c3700 1 -- 172.21.15.135:0/3538536437 >> 172.21.15.135:0/3538536437 conn(0x7f12b40f09a0 msgr2=0x7f12b40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:51.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.948+0000 7f12b89c3700 1 -- 172.21.15.135:0/3538536437 shutdown_connections 2023-11-15T06:41:51.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.948+0000 7f12b89c3700 1 -- 172.21.15.135:0/3538536437 wait complete. 2023-11-15T06:41:51.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.948+0000 7f12b89c3700 1 Processor -- start 2023-11-15T06:41:51.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.948+0000 7f12b89c3700 1 -- start start 2023-11-15T06:41:51.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.949+0000 7f12b89c3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b41053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:51.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.949+0000 7f12b89c3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f12b41058a0 con 0x7f12b40f51a0 2023-11-15T06:41:51.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.949+0000 7f12b37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b41053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:51.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.949+0000 7f12b37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b41053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56770/0 (socket says 172.21.15.135:56770) 2023-11-15T06:41:51.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.949+0000 7f12b37fe700 1 -- 172.21.15.135:0/1097334153 learned_addr learned my addr 172.21.15.135:0/1097334153 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:51.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.949+0000 7f12b37fe700 1 -- 172.21.15.135:0/1097334153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f12a4005040 con 0x7f12b40f51a0 2023-11-15T06:41:51.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.949+0000 7f12b37fe700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b41053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f12a4013510 tx=0x7f12a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:51.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.950+0000 7f129bfff700 1 -- 172.21.15.135:0/1097334153 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a4009320 con 0x7f12b40f51a0 2023-11-15T06:41:51.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.950+0000 7f129bfff700 1 -- 172.21.15.135:0/1097334153 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f12a4013f00 con 0x7f12b40f51a0 2023-11-15T06:41:51.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.950+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f12b4105aa0 con 0x7f12b40f51a0 2023-11-15T06:41:51.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.950+0000 7f129bfff700 1 -- 172.21.15.135:0/1097334153 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f12a4014dc0 con 0x7f12b40f51a0 2023-11-15T06:41:51.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.950+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f12b4105f30 con 0x7f12b40f51a0 2023-11-15T06:41:51.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.950+0000 7f129bfff700 1 -- 172.21.15.135:0/1097334153 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f12a4014420 con 0x7f12b40f51a0 2023-11-15T06:41:51.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.951+0000 7f129bfff700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f129c02ffb0 0x7f129c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:51.958 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.951+0000 7f129bfff700 1 -- 172.21.15.135:0/1097334153 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f12a40410f0 con 0x7f12b40f51a0 2023-11-15T06:41:51.959 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.951+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f12b4045ad0 con 0x7f12b40f51a0 2023-11-15T06:41:51.959 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.951+0000 7f12b2ffd700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f129c02ffb0 0x7f129c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:51.959 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.951+0000 7f12b2ffd700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f129c02ffb0 0x7f129c032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f12a80096a0 tx=0x7f12a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:51.960 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:51.955+0000 7f129bfff700 1 -- 172.21.15.135:0/1097334153 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f12a40095f0 con 0x7f12b40f51a0 2023-11-15T06:41:52.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.161+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f12b4106cf0 con 0x7f12b40f51a0 2023-11-15T06:41:52.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.162+0000 7f129bfff700 1 -- 172.21.15.135:0/1097334153 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f12a402b020 con 0x7f12b40f51a0 2023-11-15T06:41:52.165 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:52.166 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f129c02ffb0 msgr2=0x7f129c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f129c02ffb0 0x7f129c032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f12a80096a0 tx=0x7f12a8006b40).stop 2023-11-15T06:41:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 msgr2=0x7f12b41053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b41053b0 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f12a4013510 tx=0x7f12a400b040).stop 2023-11-15T06:41:52.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 shutdown_connections 2023-11-15T06:41:52.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f129c02ffb0 0x7f129c032470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:52.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 --2- 172.21.15.135:0/1097334153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f12b40f51a0 0x7f12b41053b0 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:52.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 >> 172.21.15.135:0/1097334153 conn(0x7f12b40f09a0 msgr2=0x7f12b40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:52.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 shutdown_connections 2023-11-15T06:41:52.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:52.164+0000 7f12b89c3700 1 -- 172.21.15.135:0/1097334153 wait complete. 2023-11-15T06:41:52.169 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:52.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:52.163317+0000 mon.smithi123 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.135:0/1097334153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:53.541 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:53.541 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:53.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:52.269954+0000 mgr.smithi123.razjtn (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:54.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 -- 172.21.15.135:0/1061552705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 msgr2=0x7f9dac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:54.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 --2- 172.21.15.135:0/1061552705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac0f90f0 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f9d9c009fb0 tx=0x7f9d9c009b30).stop 2023-11-15T06:41:54.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 -- 172.21.15.135:0/1061552705 shutdown_connections 2023-11-15T06:41:54.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 --2- 172.21.15.135:0/1061552705 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac0f90f0 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:54.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 -- 172.21.15.135:0/1061552705 >> 172.21.15.135:0/1061552705 conn(0x7f9dac0f0b70 msgr2=0x7f9dac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:54.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 -- 172.21.15.135:0/1061552705 shutdown_connections 2023-11-15T06:41:54.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 -- 172.21.15.135:0/1061552705 wait complete. 2023-11-15T06:41:54.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 Processor -- start 2023-11-15T06:41:54.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.792+0000 7f9db1aa2700 1 -- start start 2023-11-15T06:41:54.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9db1aa2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:54.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9db1aa2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9dac106bd0 con 0x7f9dac060280 2023-11-15T06:41:54.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9dab7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:54.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9dab7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56784/0 (socket says 172.21.15.135:56784) 2023-11-15T06:41:54.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9dab7fe700 1 -- 172.21.15.135:0/2187763736 learned_addr learned my addr 172.21.15.135:0/2187763736 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:54.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9dab7fe700 1 -- 172.21.15.135:0/2187763736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c005040 con 0x7f9dac060280 2023-11-15T06:41:54.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9dab7fe700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac1066e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9d9c013510 tx=0x7f9d9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:54.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9d93fff700 1 -- 172.21.15.135:0/2187763736 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9c009320 con 0x7f9dac060280 2023-11-15T06:41:54.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9dac106dd0 con 0x7f9dac060280 2023-11-15T06:41:54.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9d93fff700 1 -- 172.21.15.135:0/2187763736 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d9c013f00 con 0x7f9dac060280 2023-11-15T06:41:54.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9d93fff700 1 -- 172.21.15.135:0/2187763736 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9c014dc0 con 0x7f9dac060280 2023-11-15T06:41:54.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.793+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9dac107260 con 0x7f9dac060280 2023-11-15T06:41:54.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.794+0000 7f9d93fff700 1 -- 172.21.15.135:0/2187763736 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d9c014420 con 0x7f9dac060280 2023-11-15T06:41:54.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.794+0000 7f9d93fff700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9d9402ff60 0x7f9d94032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:54.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.794+0000 7f9d93fff700 1 -- 172.21.15.135:0/2187763736 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d9c041030 con 0x7f9dac060280 2023-11-15T06:41:54.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.794+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9dac045ad0 con 0x7f9dac060280 2023-11-15T06:41:54.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.795+0000 7f9daaffd700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9d9402ff60 0x7f9d94032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:54.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.795+0000 7f9daaffd700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9d9402ff60 0x7f9d94032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f9da00096a0 tx=0x7f9da0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:54.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:54.799+0000 7f9d93fff700 1 -- 172.21.15.135:0/2187763736 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d9c0095f0 con 0x7f9dac060280 2023-11-15T06:41:55.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.004+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9dac0fc040 con 0x7f9dac060280 2023-11-15T06:41:55.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.005+0000 7f9d93fff700 1 -- 172.21.15.135:0/2187763736 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d9c02b020 con 0x7f9dac060280 2023-11-15T06:41:55.008 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:55.008 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:55.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9d9402ff60 msgr2=0x7f9d94032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:55.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9d9402ff60 0x7f9d94032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f9da00096a0 tx=0x7f9da0006b40).stop 2023-11-15T06:41:55.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 msgr2=0x7f9dac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:55.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac1066e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f9d9c013510 tx=0x7f9d9c00b040).stop 2023-11-15T06:41:55.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 shutdown_connections 2023-11-15T06:41:55.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9d9402ff60 0x7f9d94032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:55.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 --2- 172.21.15.135:0/2187763736 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9dac060280 0x7f9dac1066e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:55.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 >> 172.21.15.135:0/2187763736 conn(0x7f9dac0f0b70 msgr2=0x7f9dac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:55.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 shutdown_connections 2023-11-15T06:41:55.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:55.006+0000 7f9db1aa2700 1 -- 172.21.15.135:0/2187763736 wait complete. 2023-11-15T06:41:55.012 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:55.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:54.270288+0000 mgr.smithi123.razjtn (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:55.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:55.005651+0000 mon.smithi123 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.135:0/2187763736' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:56.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:56.407 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:57.513 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.510+0000 7fb864fce700 1 -- 172.21.15.135:0/1607870313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 msgr2=0x7fb8600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:57.513 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.510+0000 7fb864fce700 1 --2- 172.21.15.135:0/1607870313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb8600f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7fb854009fb0 tx=0x7fb854009b30).stop 2023-11-15T06:41:57.513 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.510+0000 7fb864fce700 1 -- 172.21.15.135:0/1607870313 shutdown_connections 2023-11-15T06:41:57.514 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.510+0000 7fb864fce700 1 --2- 172.21.15.135:0/1607870313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb8600f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:57.514 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.510+0000 7fb864fce700 1 -- 172.21.15.135:0/1607870313 >> 172.21.15.135:0/1607870313 conn(0x7fb8600f0b70 msgr2=0x7fb8600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:57.514 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.510+0000 7fb864fce700 1 -- 172.21.15.135:0/1607870313 shutdown_connections 2023-11-15T06:41:57.514 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.510+0000 7fb864fce700 1 -- 172.21.15.135:0/1607870313 wait complete. 2023-11-15T06:41:57.516 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.511+0000 7fb864fce700 1 Processor -- start 2023-11-15T06:41:57.516 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.511+0000 7fb864fce700 1 -- start start 2023-11-15T06:41:57.517 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.511+0000 7fb864fce700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb860109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:57.517 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.511+0000 7fb864fce700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb860060220 con 0x7fb8600f5410 2023-11-15T06:41:57.517 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.511+0000 7fb85e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb860109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:57.517 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.511+0000 7fb85e59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb860109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56796/0 (socket says 172.21.15.135:56796) 2023-11-15T06:41:57.517 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.511+0000 7fb85e59c700 1 -- 172.21.15.135:0/2317832226 learned_addr learned my addr 172.21.15.135:0/2317832226 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:41:57.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.512+0000 7fb85e59c700 1 -- 172.21.15.135:0/2317832226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb854005040 con 0x7fb8600f5410 2023-11-15T06:41:57.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.512+0000 7fb85e59c700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb860109450 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb854013510 tx=0x7fb85400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:57.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.512+0000 7fb84effd700 1 -- 172.21.15.135:0/2317832226 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb854009320 con 0x7fb8600f5410 2023-11-15T06:41:57.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.512+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb860060420 con 0x7fb8600f5410 2023-11-15T06:41:57.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.512+0000 7fb84effd700 1 -- 172.21.15.135:0/2317832226 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb854013f00 con 0x7fb8600f5410 2023-11-15T06:41:57.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.512+0000 7fb84effd700 1 -- 172.21.15.135:0/2317832226 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb854014d90 con 0x7fb8600f5410 2023-11-15T06:41:57.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.512+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb860109b00 con 0x7fb8600f5410 2023-11-15T06:41:57.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.513+0000 7fb84effd700 1 -- 172.21.15.135:0/2317832226 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8540095f0 con 0x7fb8600f5410 2023-11-15T06:41:57.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.513+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb860059100 con 0x7fb8600f5410 2023-11-15T06:41:57.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.513+0000 7fb84effd700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb84802ff60 0x7fb848032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:41:57.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.513+0000 7fb84effd700 1 -- 172.21.15.135:0/2317832226 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb854041000 con 0x7fb8600f5410 2023-11-15T06:41:57.520 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.513+0000 7fb85dd9b700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb84802ff60 0x7fb848032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:41:57.520 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.513+0000 7fb85dd9b700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb84802ff60 0x7fb848032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb8500096a0 tx=0x7fb850006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:41:57.520 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.518+0000 7fb84effd700 1 -- 172.21.15.135:0/2317832226 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb854014420 con 0x7fb8600f5410 2023-11-15T06:41:57.728 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.725+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb860045ad0 con 0x7fb8600f5410 2023-11-15T06:41:57.729 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.726+0000 7fb84effd700 1 -- 172.21.15.135:0/2317832226 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb854010230 con 0x7fb8600f5410 2023-11-15T06:41:57.729 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:41:57.729 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:41:57.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb84802ff60 msgr2=0x7fb848032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:57.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb84802ff60 0x7fb848032420 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7fb8500096a0 tx=0x7fb850006b40).stop 2023-11-15T06:41:57.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 msgr2=0x7fb860109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:41:57.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb860109450 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7fb854013510 tx=0x7fb85400b040).stop 2023-11-15T06:41:57.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 shutdown_connections 2023-11-15T06:41:57.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb84802ff60 0x7fb848032420 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:57.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 --2- 172.21.15.135:0/2317832226 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb8600f5410 0x7fb860109450 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:41:57.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 >> 172.21.15.135:0/2317832226 conn(0x7fb8600f0b70 msgr2=0x7fb8600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:41:57.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 shutdown_connections 2023-11-15T06:41:57.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:41:57.727+0000 7fb864fce700 1 -- 172.21.15.135:0/2317832226 wait complete. 2023-11-15T06:41:57.732 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:41:57.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:56.270654+0000 mgr.smithi123.razjtn (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:41:58.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:41:57.726863+0000 mon.smithi123 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.135:0/2317832226' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:41:59.107 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:41:59.108 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:41:59.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:41:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:41:58.270925+0000 mgr.smithi123.razjtn (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:00.285 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 -- 172.21.15.135:0/1722532338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 msgr2=0x7f66440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:00.286 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 --2- 172.21.15.135:0/1722532338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f66440f5830 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f662c009fb0 tx=0x7f662c009b30).stop 2023-11-15T06:42:00.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 -- 172.21.15.135:0/1722532338 shutdown_connections 2023-11-15T06:42:00.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 --2- 172.21.15.135:0/1722532338 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f66440f5830 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:00.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 -- 172.21.15.135:0/1722532338 >> 172.21.15.135:0/1722532338 conn(0x7f66440f0b70 msgr2=0x7f66440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:00.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 -- 172.21.15.135:0/1722532338 shutdown_connections 2023-11-15T06:42:00.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 -- 172.21.15.135:0/1722532338 wait complete. 2023-11-15T06:42:00.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.282+0000 7f6649a6b700 1 Processor -- start 2023-11-15T06:42:00.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f6649a6b700 1 -- start start 2023-11-15T06:42:00.288 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f6649a6b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f6644108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:00.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f6649a6b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6644108e20 con 0x7f66440f5410 2023-11-15T06:42:00.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f66437fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f6644108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:00.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f66437fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f6644108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37530/0 (socket says 172.21.15.135:37530) 2023-11-15T06:42:00.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f66437fe700 1 -- 172.21.15.135:0/1311347137 learned_addr learned my addr 172.21.15.135:0/1311347137 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:00.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f66437fe700 1 -- 172.21.15.135:0/1311347137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f662c005040 con 0x7f66440f5410 2023-11-15T06:42:00.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.283+0000 7f66437fe700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f6644108930 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f662c013510 tx=0x7f662c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:00.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.284+0000 7f663bfff700 1 -- 172.21.15.135:0/1311347137 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f662c009320 con 0x7f66440f5410 2023-11-15T06:42:00.292 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.284+0000 7f663bfff700 1 -- 172.21.15.135:0/1311347137 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f662c013f00 con 0x7f66440f5410 2023-11-15T06:42:00.292 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.284+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6644109020 con 0x7f66440f5410 2023-11-15T06:42:00.292 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.284+0000 7f663bfff700 1 -- 172.21.15.135:0/1311347137 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f662c014d90 con 0x7f66440f5410 2023-11-15T06:42:00.292 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.284+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66441094b0 con 0x7f66440f5410 2023-11-15T06:42:00.292 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.284+0000 7f663bfff700 1 -- 172.21.15.135:0/1311347137 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f662c014420 con 0x7f66440f5410 2023-11-15T06:42:00.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.284+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6644045ad0 con 0x7f66440f5410 2023-11-15T06:42:00.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.285+0000 7f663bfff700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f663002ff50 0x7f6630032410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:00.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.285+0000 7f663bfff700 1 -- 172.21.15.135:0/1311347137 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f662c041000 con 0x7f66440f5410 2023-11-15T06:42:00.293 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.285+0000 7f6642ffd700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f663002ff50 0x7f6630032410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:00.294 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.285+0000 7f6642ffd700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f663002ff50 0x7f6630032410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f66340096a0 tx=0x7f6634006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:00.294 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.289+0000 7f663bfff700 1 -- 172.21.15.135:0/1311347137 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f662c0095f0 con 0x7f66440f5410 2023-11-15T06:42:00.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.496+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66440f9d80 con 0x7f66440f5410 2023-11-15T06:42:00.499 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.496+0000 7f663bfff700 1 -- 172.21.15.135:0/1311347137 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f662c01db70 con 0x7f66440f5410 2023-11-15T06:42:00.500 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:00.500 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:00.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f663002ff50 msgr2=0x7f6630032410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:00.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f663002ff50 0x7f6630032410 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f66340096a0 tx=0x7f6634006b40).stop 2023-11-15T06:42:00.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 msgr2=0x7f6644108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:00.501 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f6644108930 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f662c013510 tx=0x7f662c00b040).stop 2023-11-15T06:42:00.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 shutdown_connections 2023-11-15T06:42:00.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f663002ff50 0x7f6630032410 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:00.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 --2- 172.21.15.135:0/1311347137 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66440f5410 0x7f6644108930 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:00.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 >> 172.21.15.135:0/1311347137 conn(0x7f66440f0b70 msgr2=0x7f66440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:00.502 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 shutdown_connections 2023-11-15T06:42:00.503 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:00.498+0000 7f6649a6b700 1 -- 172.21.15.135:0/1311347137 wait complete. 2023-11-15T06:42:00.503 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:00.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:00 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:00.497428+0000 mon.smithi123 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.135:0/1311347137' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:01.885 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:01.886 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:01.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:00.271249+0000 mgr.smithi123.razjtn (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:02.999 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 -- 172.21.15.135:0/2100740458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 msgr2=0x7f5c600f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:02.999 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 --2- 172.21.15.135:0/2100740458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c600f55c0 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f5c5c009fb0 tx=0x7f5c5c009b30).stop 2023-11-15T06:42:03.000 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 -- 172.21.15.135:0/2100740458 shutdown_connections 2023-11-15T06:42:03.000 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 --2- 172.21.15.135:0/2100740458 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c600f55c0 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:03.000 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 -- 172.21.15.135:0/2100740458 >> 172.21.15.135:0/2100740458 conn(0x7f5c600f09a0 msgr2=0x7f5c600f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:03.001 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 -- 172.21.15.135:0/2100740458 shutdown_connections 2023-11-15T06:42:03.001 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 -- 172.21.15.135:0/2100740458 wait complete. 2023-11-15T06:42:03.005 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.996+0000 7f5c665c3700 1 Processor -- start 2023-11-15T06:42:03.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c665c3700 1 -- start start 2023-11-15T06:42:03.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c665c3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c601053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:03.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c665c3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5c601058a0 con 0x7f5c600f51a0 2023-11-15T06:42:03.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c655c1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c601053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:03.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c655c1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c601053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37552/0 (socket says 172.21.15.135:37552) 2023-11-15T06:42:03.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c655c1700 1 -- 172.21.15.135:0/2149551098 learned_addr learned my addr 172.21.15.135:0/2149551098 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:03.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c655c1700 1 -- 172.21.15.135:0/2149551098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5c5c005040 con 0x7f5c600f51a0 2023-11-15T06:42:03.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.997+0000 7f5c655c1700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c601053b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f5c5c013510 tx=0x7f5c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:03.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.998+0000 7f5c55ffb700 1 -- 172.21.15.135:0/2149551098 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c5c009320 con 0x7f5c600f51a0 2023-11-15T06:42:03.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.998+0000 7f5c55ffb700 1 -- 172.21.15.135:0/2149551098 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5c5c013f00 con 0x7f5c600f51a0 2023-11-15T06:42:03.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.998+0000 7f5c55ffb700 1 -- 172.21.15.135:0/2149551098 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5c5c014dc0 con 0x7f5c600f51a0 2023-11-15T06:42:03.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.998+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5c60105aa0 con 0x7f5c600f51a0 2023-11-15T06:42:03.009 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.998+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5c60105f30 con 0x7f5c600f51a0 2023-11-15T06:42:03.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.999+0000 7f5c55ffb700 1 -- 172.21.15.135:0/2149551098 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5c5c0095f0 con 0x7f5c600f51a0 2023-11-15T06:42:03.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.999+0000 7f5c55ffb700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5c4c02ff60 0x7f5c4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:03.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.999+0000 7f5c55ffb700 1 -- 172.21.15.135:0/2149551098 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5c5c041040 con 0x7f5c600f51a0 2023-11-15T06:42:03.010 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.999+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5c60045ad0 con 0x7f5c600f51a0 2023-11-15T06:42:03.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:02.999+0000 7f5c64dc0700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5c4c02ff60 0x7f5c4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:03.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.000+0000 7f5c64dc0700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5c4c02ff60 0x7f5c4c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5c500096a0 tx=0x7f5c50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:03.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.004+0000 7f5c55ffb700 1 -- 172.21.15.135:0/2149551098 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5c5c014420 con 0x7f5c600f51a0 2023-11-15T06:42:03.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.210+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5c600facd0 con 0x7f5c600f51a0 2023-11-15T06:42:03.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.211+0000 7f5c55ffb700 1 -- 172.21.15.135:0/2149551098 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5c5c02b020 con 0x7f5c600f51a0 2023-11-15T06:42:03.214 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:03.215 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:03.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.212+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5c4c02ff60 msgr2=0x7f5c4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:03.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.212+0000 7f5c665c3700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5c4c02ff60 0x7f5c4c032420 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f5c500096a0 tx=0x7f5c50006b40).stop 2023-11-15T06:42:03.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.212+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 msgr2=0x7f5c601053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:03.217 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.212+0000 7f5c665c3700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c601053b0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f5c5c013510 tx=0x7f5c5c00b040).stop 2023-11-15T06:42:03.217 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.213+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 shutdown_connections 2023-11-15T06:42:03.217 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.213+0000 7f5c665c3700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f5c4c02ff60 0x7f5c4c032420 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:03.218 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.213+0000 7f5c665c3700 1 --2- 172.21.15.135:0/2149551098 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f5c600f51a0 0x7f5c601053b0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:03.218 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.213+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 >> 172.21.15.135:0/2149551098 conn(0x7f5c600f09a0 msgr2=0x7f5c600f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:03.218 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.213+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 shutdown_connections 2023-11-15T06:42:03.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:03.213+0000 7f5c665c3700 1 -- 172.21.15.135:0/2149551098 wait complete. 2023-11-15T06:42:03.219 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:02.271613+0000 mgr.smithi123.razjtn (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:03.212069+0000 mon.smithi123 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.135:0/2149551098' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:04.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:04.596 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:05.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.685+0000 7f2e2ab88700 1 -- 172.21.15.135:0/1771594416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 msgr2=0x7f2e240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:05.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.685+0000 7f2e2ab88700 1 --2- 172.21.15.135:0/1771594416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e240f5830 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7f2e18009fb0 tx=0x7f2e18009b30).stop 2023-11-15T06:42:05.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.685+0000 7f2e2ab88700 1 -- 172.21.15.135:0/1771594416 shutdown_connections 2023-11-15T06:42:05.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.685+0000 7f2e2ab88700 1 --2- 172.21.15.135:0/1771594416 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e240f5830 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:05.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.685+0000 7f2e2ab88700 1 -- 172.21.15.135:0/1771594416 >> 172.21.15.135:0/1771594416 conn(0x7f2e240f0b70 msgr2=0x7f2e240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:05.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.685+0000 7f2e2ab88700 1 -- 172.21.15.135:0/1771594416 shutdown_connections 2023-11-15T06:42:05.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.685+0000 7f2e2ab88700 1 -- 172.21.15.135:0/1771594416 wait complete. 2023-11-15T06:42:05.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.686+0000 7f2e2ab88700 1 Processor -- start 2023-11-15T06:42:05.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.686+0000 7f2e2ab88700 1 -- start start 2023-11-15T06:42:05.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e2ab88700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e24109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:05.692 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e2ab88700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2e24060240 con 0x7f2e240f5410 2023-11-15T06:42:05.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e28924700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e24109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:05.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e28924700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e24109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:37566/0 (socket says 172.21.15.135:37566) 2023-11-15T06:42:05.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e28924700 1 -- 172.21.15.135:0/2553236056 learned_addr learned my addr 172.21.15.135:0/2553236056 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:05.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e28924700 1 -- 172.21.15.135:0/2553236056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2e18005040 con 0x7f2e240f5410 2023-11-15T06:42:05.693 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e28924700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e24109400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f2e18013510 tx=0x7f2e1800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:05.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e217fa700 1 -- 172.21.15.135:0/2553236056 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e18009320 con 0x7f2e240f5410 2023-11-15T06:42:05.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e217fa700 1 -- 172.21.15.135:0/2553236056 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2e18013f00 con 0x7f2e240f5410 2023-11-15T06:42:05.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2e24060440 con 0x7f2e240f5410 2023-11-15T06:42:05.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.687+0000 7f2e217fa700 1 -- 172.21.15.135:0/2553236056 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2e18014d90 con 0x7f2e240f5410 2023-11-15T06:42:05.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.688+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2e24109ab0 con 0x7f2e240f5410 2023-11-15T06:42:05.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.688+0000 7f2e217fa700 1 -- 172.21.15.135:0/2553236056 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2e180263a0 con 0x7f2e240f5410 2023-11-15T06:42:05.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.688+0000 7f2e217fa700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2e1402ff60 0x7f2e14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:05.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.688+0000 7f2e217fa700 1 -- 172.21.15.135:0/2553236056 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2e180410c0 con 0x7f2e240f5410 2023-11-15T06:42:05.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.688+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2e24045ad0 con 0x7f2e240f5410 2023-11-15T06:42:05.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.689+0000 7f2e23fff700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2e1402ff60 0x7f2e14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:05.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.689+0000 7f2e23fff700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2e1402ff60 0x7f2e14032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f2e100096a0 tx=0x7f2e10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:05.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.693+0000 7f2e217fa700 1 -- 172.21.15.135:0/2553236056 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2e18012920 con 0x7f2e240f5410 2023-11-15T06:42:05.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.902+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2e2410a190 con 0x7f2e240f5410 2023-11-15T06:42:05.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:04.271978+0000 mgr.smithi123.razjtn (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:05.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.903+0000 7f2e217fa700 1 -- 172.21.15.135:0/2553236056 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2e1802b020 con 0x7f2e240f5410 2023-11-15T06:42:05.907 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:05.907 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:05.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2e1402ff60 msgr2=0x7f2e14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:05.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2e1402ff60 0x7f2e14032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7f2e100096a0 tx=0x7f2e10006b40).stop 2023-11-15T06:42:05.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 msgr2=0x7f2e24109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:05.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e24109400 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7f2e18013510 tx=0x7f2e1800b040).stop 2023-11-15T06:42:05.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 shutdown_connections 2023-11-15T06:42:05.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2e1402ff60 0x7f2e14032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:05.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 --2- 172.21.15.135:0/2553236056 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2e240f5410 0x7f2e24109400 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:05.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 >> 172.21.15.135:0/2553236056 conn(0x7f2e240f0b70 msgr2=0x7f2e240f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:05.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 shutdown_connections 2023-11-15T06:42:05.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:05.905+0000 7f2e2ab88700 1 -- 172.21.15.135:0/2553236056 wait complete. 2023-11-15T06:42:05.910 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:06.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:06 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:05.904334+0000 mon.smithi123 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.135:0/2553236056' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:07.311 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:07.311 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:07.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:06.272321+0000 mgr.smithi123.razjtn (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:08.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 -- 172.21.15.135:0/3183272130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 msgr2=0x7f76f80f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:08.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 --2- 172.21.15.135:0/3183272130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f80f7d00 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f76e8009fb0 tx=0x7f76e8009b30).stop 2023-11-15T06:42:08.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 -- 172.21.15.135:0/3183272130 shutdown_connections 2023-11-15T06:42:08.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 --2- 172.21.15.135:0/3183272130 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f80f7d00 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:08.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 -- 172.21.15.135:0/3183272130 >> 172.21.15.135:0/3183272130 conn(0x7f76f80f0120 msgr2=0x7f76f80f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:08.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 -- 172.21.15.135:0/3183272130 shutdown_connections 2023-11-15T06:42:08.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 -- 172.21.15.135:0/3183272130 wait complete. 2023-11-15T06:42:08.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 Processor -- start 2023-11-15T06:42:08.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.451+0000 7f76f759e700 1 -- start start 2023-11-15T06:42:08.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f759e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f81064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:08.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f759e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76f81069d0 con 0x7f76f80f78e0 2023-11-15T06:42:08.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f81064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:08.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f81064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51804/0 (socket says 172.21.15.135:51804) 2023-11-15T06:42:08.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f659c700 1 -- 172.21.15.135:0/287510354 learned_addr learned my addr 172.21.15.135:0/287510354 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:08.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f659c700 1 -- 172.21.15.135:0/287510354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76e8005040 con 0x7f76f80f78e0 2023-11-15T06:42:08.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f659c700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f81064e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f76e8013510 tx=0x7f76e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:08.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76e6ffd700 1 -- 172.21.15.135:0/287510354 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e8009320 con 0x7f76f80f78e0 2023-11-15T06:42:08.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76f8106bd0 con 0x7f76f80f78e0 2023-11-15T06:42:08.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76e6ffd700 1 -- 172.21.15.135:0/287510354 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76e8013f00 con 0x7f76f80f78e0 2023-11-15T06:42:08.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.452+0000 7f76e6ffd700 1 -- 172.21.15.135:0/287510354 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76e8014dc0 con 0x7f76f80f78e0 2023-11-15T06:42:08.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.453+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76f8107060 con 0x7f76f80f78e0 2023-11-15T06:42:08.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.453+0000 7f76e6ffd700 1 -- 172.21.15.135:0/287510354 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76e80095f0 con 0x7f76f80f78e0 2023-11-15T06:42:08.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.453+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76f8045ad0 con 0x7f76f80f78e0 2023-11-15T06:42:08.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.453+0000 7f76e6ffd700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e002ff60 0x7f76e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:08.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.454+0000 7f76e6ffd700 1 -- 172.21.15.135:0/287510354 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76e80410e0 con 0x7f76f80f78e0 2023-11-15T06:42:08.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.454+0000 7f76f5d9b700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e002ff60 0x7f76e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:08.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.454+0000 7f76f5d9b700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e002ff60 0x7f76e0032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f76ec0096a0 tx=0x7f76ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:08.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.458+0000 7f76e6ffd700 1 -- 172.21.15.135:0/287510354 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76e8014420 con 0x7f76f80f78e0 2023-11-15T06:42:08.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.664+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76f8107eb0 con 0x7f76f80f78e0 2023-11-15T06:42:08.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.666+0000 7f76e6ffd700 1 -- 172.21.15.135:0/287510354 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76e801c920 con 0x7f76f80f78e0 2023-11-15T06:42:08.678 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:08.679 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:08.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e002ff60 msgr2=0x7f76e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:08.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e002ff60 0x7f76e0032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f76ec0096a0 tx=0x7f76ec006b40).stop 2023-11-15T06:42:08.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 msgr2=0x7f76f81064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:08.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f81064e0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f76e8013510 tx=0x7f76e800b040).stop 2023-11-15T06:42:08.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 shutdown_connections 2023-11-15T06:42:08.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f76e002ff60 0x7f76e0032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:08.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 --2- 172.21.15.135:0/287510354 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76f80f78e0 0x7f76f81064e0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:08.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 >> 172.21.15.135:0/287510354 conn(0x7f76f80f0120 msgr2=0x7f76f80f0ba0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:08.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 shutdown_connections 2023-11-15T06:42:08.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:08.667+0000 7f76f759e700 1 -- 172.21.15.135:0/287510354 wait complete. 2023-11-15T06:42:08.681 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:08.272691+0000 mgr.smithi123.razjtn (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:08.666084+0000 mon.smithi123 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.135:0/287510354' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:10.080 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:10.081 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:11.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.200+0000 7f6c4b507700 1 -- 172.21.15.135:0/2607151697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 msgr2=0x7f6c440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:11.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.200+0000 7f6c4b507700 1 --2- 172.21.15.135:0/2607151697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c440f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f6c34009fb0 tx=0x7f6c34009b30).stop 2023-11-15T06:42:11.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.200+0000 7f6c4b507700 1 -- 172.21.15.135:0/2607151697 shutdown_connections 2023-11-15T06:42:11.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.200+0000 7f6c4b507700 1 --2- 172.21.15.135:0/2607151697 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c440f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:11.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.200+0000 7f6c4b507700 1 -- 172.21.15.135:0/2607151697 >> 172.21.15.135:0/2607151697 conn(0x7f6c440f0b70 msgr2=0x7f6c440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:11.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.200+0000 7f6c4b507700 1 -- 172.21.15.135:0/2607151697 shutdown_connections 2023-11-15T06:42:11.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.200+0000 7f6c4b507700 1 -- 172.21.15.135:0/2607151697 wait complete. 2023-11-15T06:42:11.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.201+0000 7f6c4b507700 1 Processor -- start 2023-11-15T06:42:11.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.201+0000 7f6c4b507700 1 -- start start 2023-11-15T06:42:11.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.201+0000 7f6c4b507700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c44109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:11.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.201+0000 7f6c4b507700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c44060240 con 0x7f6c440f5410 2023-11-15T06:42:11.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.201+0000 7f6c492a3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c44109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:11.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.201+0000 7f6c492a3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c44109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51830/0 (socket says 172.21.15.135:51830) 2023-11-15T06:42:11.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.201+0000 7f6c492a3700 1 -- 172.21.15.135:0/1050937078 learned_addr learned my addr 172.21.15.135:0/1050937078 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:11.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.202+0000 7f6c492a3700 1 -- 172.21.15.135:0/1050937078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c34005040 con 0x7f6c440f5410 2023-11-15T06:42:11.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.202+0000 7f6c492a3700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c44109430 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6c34013510 tx=0x7f6c3400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:11.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.202+0000 7f6c39ffb700 1 -- 172.21.15.135:0/1050937078 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c34009320 con 0x7f6c440f5410 2023-11-15T06:42:11.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.202+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c44060440 con 0x7f6c440f5410 2023-11-15T06:42:11.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.202+0000 7f6c39ffb700 1 -- 172.21.15.135:0/1050937078 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c34013f00 con 0x7f6c440f5410 2023-11-15T06:42:11.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.202+0000 7f6c39ffb700 1 -- 172.21.15.135:0/1050937078 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c34014d90 con 0x7f6c440f5410 2023-11-15T06:42:11.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.202+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c44109ae0 con 0x7f6c440f5410 2023-11-15T06:42:11.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.203+0000 7f6c39ffb700 1 -- 172.21.15.135:0/1050937078 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c34014420 con 0x7f6c440f5410 2023-11-15T06:42:11.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.203+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c44045ad0 con 0x7f6c440f5410 2023-11-15T06:42:11.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.203+0000 7f6c39ffb700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c3002ff60 0x7f6c30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:11.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.203+0000 7f6c39ffb700 1 -- 172.21.15.135:0/1050937078 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c34041080 con 0x7f6c440f5410 2023-11-15T06:42:11.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.203+0000 7f6c48aa2700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c3002ff60 0x7f6c30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:11.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.204+0000 7f6c48aa2700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c3002ff60 0x7f6c30032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f6c400096a0 tx=0x7f6c40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:11.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.208+0000 7f6c39ffb700 1 -- 172.21.15.135:0/1050937078 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c340095f0 con 0x7f6c440f5410 2023-11-15T06:42:11.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.413+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c440fae60 con 0x7f6c440f5410 2023-11-15T06:42:11.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.414+0000 7f6c39ffb700 1 -- 172.21.15.135:0/1050937078 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6c3401db70 con 0x7f6c440f5410 2023-11-15T06:42:11.417 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:11.417 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:11.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c3002ff60 msgr2=0x7f6c30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:11.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c3002ff60 0x7f6c30032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f6c400096a0 tx=0x7f6c40006b40).stop 2023-11-15T06:42:11.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 msgr2=0x7f6c44109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:11.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c44109430 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f6c34013510 tx=0x7f6c3400b040).stop 2023-11-15T06:42:11.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 shutdown_connections 2023-11-15T06:42:11.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c3002ff60 0x7f6c30032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:11.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 --2- 172.21.15.135:0/1050937078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c440f5410 0x7f6c44109430 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:11.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 >> 172.21.15.135:0/1050937078 conn(0x7f6c440f0b70 msgr2=0x7f6c440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:11.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 shutdown_connections 2023-11-15T06:42:11.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:11.415+0000 7f6c4b507700 1 -- 172.21.15.135:0/1050937078 wait complete. 2023-11-15T06:42:11.420 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:11.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:10.273062+0000 mgr.smithi123.razjtn (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:11.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:11.414643+0000 mon.smithi123 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.135:0/1050937078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:12.809 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:12.810 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:13.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:12.273436+0000 mgr.smithi123.razjtn (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:13.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.985+0000 7f5258acf700 1 -- 172.21.15.135:0/3658550694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 msgr2=0x7f52540f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:13.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.985+0000 7f5258acf700 1 --2- 172.21.15.135:0/3658550694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f8f20 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f523c009fb0 tx=0x7f523c009b30).stop 2023-11-15T06:42:13.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.985+0000 7f5258acf700 1 -- 172.21.15.135:0/3658550694 shutdown_connections 2023-11-15T06:42:13.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.985+0000 7f5258acf700 1 --2- 172.21.15.135:0/3658550694 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f8f20 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:13.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.985+0000 7f5258acf700 1 -- 172.21.15.135:0/3658550694 >> 172.21.15.135:0/3658550694 conn(0x7f52540f09a0 msgr2=0x7f52540f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:13.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.985+0000 7f5258acf700 1 -- 172.21.15.135:0/3658550694 shutdown_connections 2023-11-15T06:42:13.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.985+0000 7f5258acf700 1 -- 172.21.15.135:0/3658550694 wait complete. 2023-11-15T06:42:13.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f5258acf700 1 Processor -- start 2023-11-15T06:42:13.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f5258acf700 1 -- start start 2023-11-15T06:42:13.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f5258acf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f69a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:13.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f5258acf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52540f4e20 con 0x7f52540601e0 2023-11-15T06:42:13.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f52537fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f69a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:13.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f52537fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f69a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51840/0 (socket says 172.21.15.135:51840) 2023-11-15T06:42:13.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f52537fe700 1 -- 172.21.15.135:0/3521693770 learned_addr learned my addr 172.21.15.135:0/3521693770 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:13.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.986+0000 7f52537fe700 1 -- 172.21.15.135:0/3521693770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f523c005040 con 0x7f52540601e0 2023-11-15T06:42:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.987+0000 7f52537fe700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f69a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f523c013510 tx=0x7f523c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.987+0000 7f524bfff700 1 -- 172.21.15.135:0/3521693770 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f523c009320 con 0x7f52540601e0 2023-11-15T06:42:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.987+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52540f4fc0 con 0x7f52540601e0 2023-11-15T06:42:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.987+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52540f5450 con 0x7f52540601e0 2023-11-15T06:42:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.987+0000 7f524bfff700 1 -- 172.21.15.135:0/3521693770 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f523c013f00 con 0x7f52540601e0 2023-11-15T06:42:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.987+0000 7f524bfff700 1 -- 172.21.15.135:0/3521693770 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f523c014cc0 con 0x7f52540601e0 2023-11-15T06:42:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.988+0000 7f524bfff700 1 -- 172.21.15.135:0/3521693770 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f523c014420 con 0x7f52540601e0 2023-11-15T06:42:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.988+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5238004fa0 con 0x7f52540601e0 2023-11-15T06:42:13.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.988+0000 7f524bfff700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f524002ffb0 0x7f5240032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:13.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.988+0000 7f524bfff700 1 -- 172.21.15.135:0/3521693770 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f523c040f20 con 0x7f52540601e0 2023-11-15T06:42:13.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.988+0000 7f5252ffd700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f524002ffb0 0x7f5240032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:13.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.989+0000 7f5252ffd700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f524002ffb0 0x7f5240032470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f52440096a0 tx=0x7f5244006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:13.997 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:13.993+0000 7f524bfff700 1 -- 172.21.15.135:0/3521693770 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f523c0095f0 con 0x7f52540601e0 2023-11-15T06:42:14.200 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.197+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5238005670 con 0x7f52540601e0 2023-11-15T06:42:14.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.198+0000 7f524bfff700 1 -- 172.21.15.135:0/3521693770 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f523c012750 con 0x7f52540601e0 2023-11-15T06:42:14.201 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:14.202 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:14.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f524002ffb0 msgr2=0x7f5240032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:14.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f524002ffb0 0x7f5240032470 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f52440096a0 tx=0x7f5244006b40).stop 2023-11-15T06:42:14.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 msgr2=0x7f52540f69a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:14.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f69a0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f523c013510 tx=0x7f523c00b040).stop 2023-11-15T06:42:14.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 shutdown_connections 2023-11-15T06:42:14.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f524002ffb0 0x7f5240032470 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:14.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 --2- 172.21.15.135:0/3521693770 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f52540601e0 0x7f52540f69a0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:14.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 >> 172.21.15.135:0/3521693770 conn(0x7f52540f09a0 msgr2=0x7f52540f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:14.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 shutdown_connections 2023-11-15T06:42:14.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:14.200+0000 7f5258acf700 1 -- 172.21.15.135:0/3521693770 wait complete. 2023-11-15T06:42:14.205 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:14.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:14 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:14.199225+0000 mon.smithi123 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.135:0/3521693770' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:15.589 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:15.590 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:15.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:14.273802+0000 mgr.smithi123.razjtn (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:16.739 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.736+0000 7f4555647700 1 -- 172.21.15.135:0/2182031168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 msgr2=0x7f45500f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:16.739 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.736+0000 7f4555647700 1 --2- 172.21.15.135:0/2182031168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f45500f6930 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f4544009fb0 tx=0x7f4544009b30).stop 2023-11-15T06:42:16.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.736+0000 7f4555647700 1 -- 172.21.15.135:0/2182031168 shutdown_connections 2023-11-15T06:42:16.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.736+0000 7f4555647700 1 --2- 172.21.15.135:0/2182031168 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f45500f6930 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:16.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.736+0000 7f4555647700 1 -- 172.21.15.135:0/2182031168 >> 172.21.15.135:0/2182031168 conn(0x7f45500f1c70 msgr2=0x7f45500f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:16.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.736+0000 7f4555647700 1 -- 172.21.15.135:0/2182031168 shutdown_connections 2023-11-15T06:42:16.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.736+0000 7f4555647700 1 -- 172.21.15.135:0/2182031168 wait complete. 2023-11-15T06:42:16.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.737+0000 7f4555647700 1 Processor -- start 2023-11-15T06:42:16.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.737+0000 7f4555647700 1 -- start start 2023-11-15T06:42:16.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.737+0000 7f4555647700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:16.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.737+0000 7f4555647700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4550106c20 con 0x7f45500f6510 2023-11-15T06:42:16.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.737+0000 7f454effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:16.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f454effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51862/0 (socket says 172.21.15.135:51862) 2023-11-15T06:42:16.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f454effd700 1 -- 172.21.15.135:0/3155574140 learned_addr learned my addr 172.21.15.135:0/3155574140 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:16.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f454effd700 1 -- 172.21.15.135:0/3155574140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4544005040 con 0x7f45500f6510 2023-11-15T06:42:16.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f454effd700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f4550106730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4544013510 tx=0x7f454400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:16.745 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f45377fe700 1 -- 172.21.15.135:0/3155574140 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4544009320 con 0x7f45500f6510 2023-11-15T06:42:16.745 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4550106e20 con 0x7f45500f6510 2023-11-15T06:42:16.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f45377fe700 1 -- 172.21.15.135:0/3155574140 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4544013f00 con 0x7f45500f6510 2023-11-15T06:42:16.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f45377fe700 1 -- 172.21.15.135:0/3155574140 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4544014d90 con 0x7f45500f6510 2023-11-15T06:42:16.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.738+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45501072b0 con 0x7f45500f6510 2023-11-15T06:42:16.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.739+0000 7f45377fe700 1 -- 172.21.15.135:0/3155574140 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45440263a0 con 0x7f45500f6510 2023-11-15T06:42:16.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.739+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4550045ad0 con 0x7f45500f6510 2023-11-15T06:42:16.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.739+0000 7f45377fe700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f453802ff60 0x7f4538032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:16.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.739+0000 7f45377fe700 1 -- 172.21.15.135:0/3155574140 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4544041020 con 0x7f45500f6510 2023-11-15T06:42:16.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.740+0000 7f454e7fc700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f453802ff60 0x7f4538032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:16.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.740+0000 7f454e7fc700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f453802ff60 0x7f4538032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f45400096a0 tx=0x7f4540006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:16.748 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.744+0000 7f45377fe700 1 -- 172.21.15.135:0/3155574140 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f45440095f0 con 0x7f45500f6510 2023-11-15T06:42:16.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.949+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45500fbf60 con 0x7f45500f6510 2023-11-15T06:42:16.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.949+0000 7f45377fe700 1 -- 172.21.15.135:0/3155574140 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f454401db70 con 0x7f45500f6510 2023-11-15T06:42:16.953 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:16.953 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:16.954 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f453802ff60 msgr2=0x7f4538032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:16.954 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f453802ff60 0x7f4538032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f45400096a0 tx=0x7f4540006b40).stop 2023-11-15T06:42:16.954 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 msgr2=0x7f4550106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:16.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f4550106730 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f4544013510 tx=0x7f454400b040).stop 2023-11-15T06:42:16.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 shutdown_connections 2023-11-15T06:42:16.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f453802ff60 0x7f4538032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:16.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 --2- 172.21.15.135:0/3155574140 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45500f6510 0x7f4550106730 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:16.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 >> 172.21.15.135:0/3155574140 conn(0x7f45500f1c70 msgr2=0x7f45500f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:16.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 shutdown_connections 2023-11-15T06:42:16.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:16.951+0000 7f4555647700 1 -- 172.21.15.135:0/3155574140 wait complete. 2023-11-15T06:42:16.956 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:17.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:16.274061+0000 mgr.smithi123.razjtn (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:17.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:16.950564+0000 mon.smithi123 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.135:0/3155574140' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:18.317 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:18.318 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:19.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.445+0000 7f9143a81700 1 -- 172.21.15.135:0/19121612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 msgr2=0x7f913c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:19.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.445+0000 7f9143a81700 1 --2- 172.21.15.135:0/19121612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c0f5830 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7f912c009fb0 tx=0x7f912c009b30).stop 2023-11-15T06:42:19.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.445+0000 7f9143a81700 1 -- 172.21.15.135:0/19121612 shutdown_connections 2023-11-15T06:42:19.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.445+0000 7f9143a81700 1 --2- 172.21.15.135:0/19121612 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c0f5830 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:19.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.445+0000 7f9143a81700 1 -- 172.21.15.135:0/19121612 >> 172.21.15.135:0/19121612 conn(0x7f913c0f0b70 msgr2=0x7f913c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:19.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.445+0000 7f9143a81700 1 -- 172.21.15.135:0/19121612 shutdown_connections 2023-11-15T06:42:19.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.445+0000 7f9143a81700 1 -- 172.21.15.135:0/19121612 wait complete. 2023-11-15T06:42:19.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.446+0000 7f9143a81700 1 Processor -- start 2023-11-15T06:42:19.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.446+0000 7f9143a81700 1 -- start start 2023-11-15T06:42:19.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.446+0000 7f9143a81700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:19.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.446+0000 7f9143a81700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f913c060220 con 0x7f913c0f5410 2023-11-15T06:42:19.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.446+0000 7f914181d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:19.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.446+0000 7f914181d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58730/0 (socket says 172.21.15.135:58730) 2023-11-15T06:42:19.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.446+0000 7f914181d700 1 -- 172.21.15.135:0/2584505082 learned_addr learned my addr 172.21.15.135:0/2584505082 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:19.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.447+0000 7f914181d700 1 -- 172.21.15.135:0/2584505082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f912c005040 con 0x7f913c0f5410 2023-11-15T06:42:19.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.447+0000 7f914181d700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c109450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f912c013510 tx=0x7f912c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:19.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.447+0000 7f91327fc700 1 -- 172.21.15.135:0/2584505082 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f912c009320 con 0x7f913c0f5410 2023-11-15T06:42:19.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.447+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f913c060420 con 0x7f913c0f5410 2023-11-15T06:42:19.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.447+0000 7f91327fc700 1 -- 172.21.15.135:0/2584505082 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f912c013f00 con 0x7f913c0f5410 2023-11-15T06:42:19.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.447+0000 7f91327fc700 1 -- 172.21.15.135:0/2584505082 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f912c014d90 con 0x7f913c0f5410 2023-11-15T06:42:19.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.447+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f913c109b00 con 0x7f913c0f5410 2023-11-15T06:42:19.454 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.448+0000 7f91327fc700 1 -- 172.21.15.135:0/2584505082 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f912c0095f0 con 0x7f913c0f5410 2023-11-15T06:42:19.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.448+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f913c059100 con 0x7f913c0f5410 2023-11-15T06:42:19.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.448+0000 7f91327fc700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f912802ff60 0x7f9128032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:19.455 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.448+0000 7f91327fc700 1 -- 172.21.15.135:0/2584505082 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f912c040fe0 con 0x7f913c0f5410 2023-11-15T06:42:19.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.448+0000 7f914101c700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f912802ff60 0x7f9128032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:19.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.449+0000 7f914101c700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f912802ff60 0x7f9128032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f91380096a0 tx=0x7f9138006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:19.456 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.453+0000 7f91327fc700 1 -- 172.21.15.135:0/2584505082 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f912c01d7f0 con 0x7f913c0f5410 2023-11-15T06:42:19.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.661+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f913c045ad0 con 0x7f913c0f5410 2023-11-15T06:42:19.664 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.662+0000 7f91327fc700 1 -- 172.21.15.135:0/2584505082 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f912c010230 con 0x7f913c0f5410 2023-11-15T06:42:19.665 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:19.665 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:19.666 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f912802ff60 msgr2=0x7f9128032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:19.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f912802ff60 0x7f9128032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f91380096a0 tx=0x7f9138006b40).stop 2023-11-15T06:42:19.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 msgr2=0x7f913c109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:19.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c109450 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7f912c013510 tx=0x7f912c00b040).stop 2023-11-15T06:42:19.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 shutdown_connections 2023-11-15T06:42:19.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f912802ff60 0x7f9128032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:19.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 --2- 172.21.15.135:0/2584505082 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f913c0f5410 0x7f913c109450 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:19.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 >> 172.21.15.135:0/2584505082 conn(0x7f913c0f0b70 msgr2=0x7f913c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:19.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 shutdown_connections 2023-11-15T06:42:19.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:19.664+0000 7f9143a81700 1 -- 172.21.15.135:0/2584505082 wait complete. 2023-11-15T06:42:19.669 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:19.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:18.274311+0000 mgr.smithi123.razjtn (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:20.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:20 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:19.662941+0000 mon.smithi123 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.135:0/2584505082' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:21.056 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:21.057 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:21.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:20.274561+0000 mgr.smithi123.razjtn (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:22.194 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.189+0000 7ffb70d5f700 1 -- 172.21.15.135:0/2235362929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 msgr2=0x7ffb6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:22.194 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.189+0000 7ffb70d5f700 1 --2- 172.21.15.135:0/2235362929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c0f6930 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7ffb54009fb0 tx=0x7ffb54009b30).stop 2023-11-15T06:42:22.194 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 -- 172.21.15.135:0/2235362929 shutdown_connections 2023-11-15T06:42:22.195 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 --2- 172.21.15.135:0/2235362929 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c0f6930 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:22.195 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 -- 172.21.15.135:0/2235362929 >> 172.21.15.135:0/2235362929 conn(0x7ffb6c0f1c70 msgr2=0x7ffb6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:22.195 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 -- 172.21.15.135:0/2235362929 shutdown_connections 2023-11-15T06:42:22.195 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 -- 172.21.15.135:0/2235362929 wait complete. 2023-11-15T06:42:22.195 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 Processor -- start 2023-11-15T06:42:22.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 -- start start 2023-11-15T06:42:22.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:22.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb70d5f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb6c106c20 con 0x7ffb6c0f6510 2023-11-15T06:42:22.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.190+0000 7ffb6a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:22.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb6a59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58748/0 (socket says 172.21.15.135:58748) 2023-11-15T06:42:22.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb6a59c700 1 -- 172.21.15.135:0/3366254301 learned_addr learned my addr 172.21.15.135:0/3366254301 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:22.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb6a59c700 1 -- 172.21.15.135:0/3366254301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb54005040 con 0x7ffb6c0f6510 2023-11-15T06:42:22.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb6a59c700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ffb54013510 tx=0x7ffb5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:22.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb62ffd700 1 -- 172.21.15.135:0/3366254301 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb54009320 con 0x7ffb6c0f6510 2023-11-15T06:42:22.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb6c106e20 con 0x7ffb6c0f6510 2023-11-15T06:42:22.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb6c1072b0 con 0x7ffb6c0f6510 2023-11-15T06:42:22.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb62ffd700 1 -- 172.21.15.135:0/3366254301 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb54013f00 con 0x7ffb6c0f6510 2023-11-15T06:42:22.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.191+0000 7ffb62ffd700 1 -- 172.21.15.135:0/3366254301 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb54014d60 con 0x7ffb6c0f6510 2023-11-15T06:42:22.200 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.192+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb6c045ad0 con 0x7ffb6c0f6510 2023-11-15T06:42:22.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.192+0000 7ffb62ffd700 1 -- 172.21.15.135:0/3366254301 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb54014420 con 0x7ffb6c0f6510 2023-11-15T06:42:22.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.192+0000 7ffb62ffd700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5802ff60 0x7ffb58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:22.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.192+0000 7ffb62ffd700 1 -- 172.21.15.135:0/3366254301 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb540412a0 con 0x7ffb6c0f6510 2023-11-15T06:42:22.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.192+0000 7ffb69d9b700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5802ff60 0x7ffb58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:22.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.193+0000 7ffb69d9b700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5802ff60 0x7ffb58032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ffb5c0096a0 tx=0x7ffb5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:22.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.197+0000 7ffb62ffd700 1 -- 172.21.15.135:0/3366254301 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb5401c320 con 0x7ffb6c0f6510 2023-11-15T06:42:22.406 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.403+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb6c0fbf60 con 0x7ffb6c0f6510 2023-11-15T06:42:22.406 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.404+0000 7ffb62ffd700 1 -- 172.21.15.135:0/3366254301 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb54012b20 con 0x7ffb6c0f6510 2023-11-15T06:42:22.407 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:22.407 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:22.408 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5802ff60 msgr2=0x7ffb58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:22.408 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5802ff60 0x7ffb58032420 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7ffb5c0096a0 tx=0x7ffb5c006b40).stop 2023-11-15T06:42:22.409 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 msgr2=0x7ffb6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:22.409 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c106730 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7ffb54013510 tx=0x7ffb5400b040).stop 2023-11-15T06:42:22.409 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 shutdown_connections 2023-11-15T06:42:22.409 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5802ff60 0x7ffb58032420 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:22.409 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 --2- 172.21.15.135:0/3366254301 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb6c0f6510 0x7ffb6c106730 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:22.410 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 >> 172.21.15.135:0/3366254301 conn(0x7ffb6c0f1c70 msgr2=0x7ffb6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:22.410 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 shutdown_connections 2023-11-15T06:42:22.410 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:22.405+0000 7ffb70d5f700 1 -- 172.21.15.135:0/3366254301 wait complete. 2023-11-15T06:42:22.410 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:22.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:22 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:22.404829+0000 mon.smithi123 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.135:0/3366254301' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:23.794 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:23.795 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:23.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:22.274926+0000 mgr.smithi123.razjtn (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:24.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.984+0000 7fa3f02db700 1 -- 172.21.15.135:0/3098019700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 msgr2=0x7fa3e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:24.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.984+0000 7fa3f02db700 1 --2- 172.21.15.135:0/3098019700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e80f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7fa3e4009fb0 tx=0x7fa3e4009b30).stop 2023-11-15T06:42:24.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.984+0000 7fa3f02db700 1 -- 172.21.15.135:0/3098019700 shutdown_connections 2023-11-15T06:42:24.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.984+0000 7fa3f02db700 1 --2- 172.21.15.135:0/3098019700 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e80f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:24.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.984+0000 7fa3f02db700 1 -- 172.21.15.135:0/3098019700 >> 172.21.15.135:0/3098019700 conn(0x7fa3e80f1c70 msgr2=0x7fa3e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:24.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.984+0000 7fa3f02db700 1 -- 172.21.15.135:0/3098019700 shutdown_connections 2023-11-15T06:42:24.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.984+0000 7fa3f02db700 1 -- 172.21.15.135:0/3098019700 wait complete. 2023-11-15T06:42:24.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.985+0000 7fa3f02db700 1 Processor -- start 2023-11-15T06:42:24.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.985+0000 7fa3f02db700 1 -- start start 2023-11-15T06:42:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.985+0000 7fa3f02db700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.985+0000 7fa3f02db700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3e8069e20 con 0x7fa3e80f6510 2023-11-15T06:42:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.985+0000 7fa3ee077700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.985+0000 7fa3ee077700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58774/0 (socket says 172.21.15.135:58774) 2023-11-15T06:42:24.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.985+0000 7fa3ee077700 1 -- 172.21.15.135:0/2281543942 learned_addr learned my addr 172.21.15.135:0/2281543942 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.986+0000 7fa3ee077700 1 -- 172.21.15.135:0/2281543942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3e4005040 con 0x7fa3e80f6510 2023-11-15T06:42:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.986+0000 7fa3ee077700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa3e4013510 tx=0x7fa3e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.986+0000 7fa3deffd700 1 -- 172.21.15.135:0/2281543942 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3e4009320 con 0x7fa3e80f6510 2023-11-15T06:42:24.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.986+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3e8069fc0 con 0x7fa3e80f6510 2023-11-15T06:42:24.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.986+0000 7fa3deffd700 1 -- 172.21.15.135:0/2281543942 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3e4013f00 con 0x7fa3e80f6510 2023-11-15T06:42:24.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.986+0000 7fa3deffd700 1 -- 172.21.15.135:0/2281543942 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3e4014d90 con 0x7fa3e80f6510 2023-11-15T06:42:24.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.986+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3e806a450 con 0x7fa3e80f6510 2023-11-15T06:42:24.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.987+0000 7fa3deffd700 1 -- 172.21.15.135:0/2281543942 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3e40263a0 con 0x7fa3e80f6510 2023-11-15T06:42:24.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.987+0000 7fa3deffd700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa3d402ff60 0x7fa3d4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:24.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.987+0000 7fa3deffd700 1 -- 172.21.15.135:0/2281543942 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3e4041100 con 0x7fa3e80f6510 2023-11-15T06:42:24.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.987+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3e8059100 con 0x7fa3e80f6510 2023-11-15T06:42:24.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.987+0000 7fa3ed876700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa3d402ff60 0x7fa3d4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:24.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.987+0000 7fa3ed876700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa3d402ff60 0x7fa3d4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa3d80096a0 tx=0x7fa3d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:24.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:24.991+0000 7fa3deffd700 1 -- 172.21.15.135:0/2281543942 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3e4012920 con 0x7fa3e80f6510 2023-11-15T06:42:25.198 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.196+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3e80fbf60 con 0x7fa3e80f6510 2023-11-15T06:42:25.199 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.196+0000 7fa3deffd700 1 -- 172.21.15.135:0/2281543942 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa3e4012330 con 0x7fa3e80f6510 2023-11-15T06:42:25.199 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:25.200 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:25.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa3d402ff60 msgr2=0x7fa3d4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:25.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa3d402ff60 0x7fa3d4032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7fa3d80096a0 tx=0x7fa3d8006b40).stop 2023-11-15T06:42:25.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 msgr2=0x7fa3e806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:25.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7fa3e4013510 tx=0x7fa3e400b040).stop 2023-11-15T06:42:25.201 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 shutdown_connections 2023-11-15T06:42:25.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa3d402ff60 0x7fa3d4032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:25.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 --2- 172.21.15.135:0/2281543942 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa3e80f6510 0x7fa3e806b9a0 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:25.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 >> 172.21.15.135:0/2281543942 conn(0x7fa3e80f1c70 msgr2=0x7fa3e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:25.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 shutdown_connections 2023-11-15T06:42:25.202 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:25.198+0000 7fa3f02db700 1 -- 172.21.15.135:0/2281543942 wait complete. 2023-11-15T06:42:25.203 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:24.275265+0000 mgr.smithi123.razjtn (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:25.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:25.197498+0000 mon.smithi123 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.135:0/2281543942' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:26.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:26.587 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:27.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.690+0000 7f6c860fe700 1 -- 172.21.15.135:0/1675094312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 msgr2=0x7f6c800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:27.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.690+0000 7f6c860fe700 1 --2- 172.21.15.135:0/1675094312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c800f55c0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7f6c7c009fb0 tx=0x7f6c7c009b30).stop 2023-11-15T06:42:27.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.691+0000 7f6c860fe700 1 -- 172.21.15.135:0/1675094312 shutdown_connections 2023-11-15T06:42:27.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.691+0000 7f6c860fe700 1 --2- 172.21.15.135:0/1675094312 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c800f55c0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:27.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.691+0000 7f6c860fe700 1 -- 172.21.15.135:0/1675094312 >> 172.21.15.135:0/1675094312 conn(0x7f6c800f09a0 msgr2=0x7f6c800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:27.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.691+0000 7f6c860fe700 1 -- 172.21.15.135:0/1675094312 shutdown_connections 2023-11-15T06:42:27.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.691+0000 7f6c860fe700 1 -- 172.21.15.135:0/1675094312 wait complete. 2023-11-15T06:42:27.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.691+0000 7f6c860fe700 1 Processor -- start 2023-11-15T06:42:27.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.691+0000 7f6c860fe700 1 -- start start 2023-11-15T06:42:27.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c860fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c801053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:27.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c860fe700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c801058a0 con 0x7f6c800f51a0 2023-11-15T06:42:27.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c850fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c801053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:27.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c850fc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c801053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58790/0 (socket says 172.21.15.135:58790) 2023-11-15T06:42:27.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c850fc700 1 -- 172.21.15.135:0/3896860063 learned_addr learned my addr 172.21.15.135:0/3896860063 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:27.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c850fc700 1 -- 172.21.15.135:0/3896860063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c7c005040 con 0x7f6c800f51a0 2023-11-15T06:42:27.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c850fc700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c801053b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f6c7c013510 tx=0x7f6c7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:27.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c75ffb700 1 -- 172.21.15.135:0/3896860063 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7c009320 con 0x7f6c800f51a0 2023-11-15T06:42:27.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c80105aa0 con 0x7f6c800f51a0 2023-11-15T06:42:27.699 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c75ffb700 1 -- 172.21.15.135:0/3896860063 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6c7c013f00 con 0x7f6c800f51a0 2023-11-15T06:42:27.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.692+0000 7f6c75ffb700 1 -- 172.21.15.135:0/3896860063 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c7c014dc0 con 0x7f6c800f51a0 2023-11-15T06:42:27.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.693+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c80105f30 con 0x7f6c800f51a0 2023-11-15T06:42:27.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.693+0000 7f6c75ffb700 1 -- 172.21.15.135:0/3896860063 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6c7c01d3d0 con 0x7f6c800f51a0 2023-11-15T06:42:27.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.693+0000 7f6c75ffb700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c6c02ff60 0x7f6c6c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:27.700 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.694+0000 7f6c75ffb700 1 -- 172.21.15.135:0/3896860063 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6c7c0410e0 con 0x7f6c800f51a0 2023-11-15T06:42:27.701 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.694+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c80045ad0 con 0x7f6c800f51a0 2023-11-15T06:42:27.701 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.694+0000 7f6c848fb700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c6c02ff60 0x7f6c6c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:27.702 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.694+0000 7f6c848fb700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c6c02ff60 0x7f6c6c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6c700096a0 tx=0x7f6c70006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:27.702 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.698+0000 7f6c75ffb700 1 -- 172.21.15.135:0/3896860063 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c7c012920 con 0x7f6c800f51a0 2023-11-15T06:42:27.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:26.275583+0000 mgr.smithi123.razjtn (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:27.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.904+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6c80106c50 con 0x7f6c800f51a0 2023-11-15T06:42:27.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.905+0000 7f6c75ffb700 1 -- 172.21.15.135:0/3896860063 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6c7c02b020 con 0x7f6c800f51a0 2023-11-15T06:42:27.908 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:27.908 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:27.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.906+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c6c02ff60 msgr2=0x7f6c6c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:27.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.906+0000 7f6c860fe700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c6c02ff60 0x7f6c6c032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f6c700096a0 tx=0x7f6c70006b40).stop 2023-11-15T06:42:27.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.906+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 msgr2=0x7f6c801053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:27.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.906+0000 7f6c860fe700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c801053b0 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7f6c7c013510 tx=0x7f6c7c00b040).stop 2023-11-15T06:42:27.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.907+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 shutdown_connections 2023-11-15T06:42:27.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.907+0000 7f6c860fe700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6c6c02ff60 0x7f6c6c032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:27.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.907+0000 7f6c860fe700 1 --2- 172.21.15.135:0/3896860063 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6c800f51a0 0x7f6c801053b0 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:27.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.907+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 >> 172.21.15.135:0/3896860063 conn(0x7f6c800f09a0 msgr2=0x7f6c800f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:27.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.907+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 shutdown_connections 2023-11-15T06:42:27.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:27.907+0000 7f6c860fe700 1 -- 172.21.15.135:0/3896860063 wait complete. 2023-11-15T06:42:27.912 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:28.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:28 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:27.906009+0000 mon.smithi123 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.135:0/3896860063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:29.295 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:29.295 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:29.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:28.275868+0000 mgr.smithi123.razjtn (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:30.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 -- 172.21.15.135:0/3813061801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 msgr2=0x7fcad80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:30.420 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 --2- 172.21.15.135:0/3813061801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad80f6930 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7fcacc009fb0 tx=0x7fcacc009b30).stop 2023-11-15T06:42:30.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 -- 172.21.15.135:0/3813061801 shutdown_connections 2023-11-15T06:42:30.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 --2- 172.21.15.135:0/3813061801 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad80f6930 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:30.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 -- 172.21.15.135:0/3813061801 >> 172.21.15.135:0/3813061801 conn(0x7fcad80f1c70 msgr2=0x7fcad80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:30.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 -- 172.21.15.135:0/3813061801 shutdown_connections 2023-11-15T06:42:30.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 -- 172.21.15.135:0/3813061801 wait complete. 2023-11-15T06:42:30.422 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 Processor -- start 2023-11-15T06:42:30.422 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.417+0000 7fcadeaec700 1 -- start start 2023-11-15T06:42:30.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.418+0000 7fcadeaec700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:30.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.418+0000 7fcadeaec700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcad8069e20 con 0x7fcad80f6510 2023-11-15T06:42:30.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.418+0000 7fcadc888700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:30.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.418+0000 7fcadc888700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:52016/0 (socket says 172.21.15.135:52016) 2023-11-15T06:42:30.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.418+0000 7fcadc888700 1 -- 172.21.15.135:0/821455729 learned_addr learned my addr 172.21.15.135:0/821455729 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:30.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.418+0000 7fcadc888700 1 -- 172.21.15.135:0/821455729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcacc005040 con 0x7fcad80f6510 2023-11-15T06:42:30.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.418+0000 7fcadc888700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fcacc013510 tx=0x7fcacc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:30.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.419+0000 7fcad57fa700 1 -- 172.21.15.135:0/821455729 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcacc009320 con 0x7fcad80f6510 2023-11-15T06:42:30.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.419+0000 7fcad57fa700 1 -- 172.21.15.135:0/821455729 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcacc013f00 con 0x7fcad80f6510 2023-11-15T06:42:30.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.419+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcad8069fc0 con 0x7fcad80f6510 2023-11-15T06:42:30.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.419+0000 7fcad57fa700 1 -- 172.21.15.135:0/821455729 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcacc014d90 con 0x7fcad80f6510 2023-11-15T06:42:30.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.419+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcad806a450 con 0x7fcad80f6510 2023-11-15T06:42:30.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.419+0000 7fcad57fa700 1 -- 172.21.15.135:0/821455729 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcacc0263a0 con 0x7fcad80f6510 2023-11-15T06:42:30.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.420+0000 7fcad57fa700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcac802ff60 0x7fcac8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:30.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.420+0000 7fcad57fa700 1 -- 172.21.15.135:0/821455729 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcacc040ff0 con 0x7fcad80f6510 2023-11-15T06:42:30.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.420+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcad8059100 con 0x7fcad80f6510 2023-11-15T06:42:30.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.420+0000 7fcad7fff700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcac802ff60 0x7fcac8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:30.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.420+0000 7fcad7fff700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcac802ff60 0x7fcac8032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fcac40096a0 tx=0x7fcac4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:30.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.424+0000 7fcad57fa700 1 -- 172.21.15.135:0/821455729 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcacc012920 con 0x7fcad80f6510 2023-11-15T06:42:30.635 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.632+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcad80fbf60 con 0x7fcad80f6510 2023-11-15T06:42:30.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.633+0000 7fcad57fa700 1 -- 172.21.15.135:0/821455729 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcacc02b020 con 0x7fcad80f6510 2023-11-15T06:42:30.636 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:30.636 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:30.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.634+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcac802ff60 msgr2=0x7fcac8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:30.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.634+0000 7fcadeaec700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcac802ff60 0x7fcac8032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7fcac40096a0 tx=0x7fcac4006b40).stop 2023-11-15T06:42:30.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.634+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 msgr2=0x7fcad806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:30.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.634+0000 7fcadeaec700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7fcacc013510 tx=0x7fcacc00b040).stop 2023-11-15T06:42:30.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.635+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 shutdown_connections 2023-11-15T06:42:30.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.635+0000 7fcadeaec700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcac802ff60 0x7fcac8032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:30.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.635+0000 7fcadeaec700 1 --2- 172.21.15.135:0/821455729 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcad80f6510 0x7fcad806b9a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:30.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.635+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 >> 172.21.15.135:0/821455729 conn(0x7fcad80f1c70 msgr2=0x7fcad80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:30.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.635+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 shutdown_connections 2023-11-15T06:42:30.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:30.635+0000 7fcadeaec700 1 -- 172.21.15.135:0/821455729 wait complete. 2023-11-15T06:42:30.639 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:31.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:30.276221+0000 mgr.smithi123.razjtn (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:31.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:30.634109+0000 mon.smithi123 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.135:0/821455729' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:32.048 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:32.049 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:33.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.164+0000 7fcbfd59c700 1 -- 172.21.15.135:0/1819956635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 msgr2=0x7fcbf80f4ca0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:33.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.164+0000 7fcbfd59c700 1 --2- 172.21.15.135:0/1819956635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f4ca0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7fcbec009fb0 tx=0x7fcbec009b30).stop 2023-11-15T06:42:33.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.164+0000 7fcbfd59c700 1 -- 172.21.15.135:0/1819956635 shutdown_connections 2023-11-15T06:42:33.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.164+0000 7fcbfd59c700 1 --2- 172.21.15.135:0/1819956635 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f4ca0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:33.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.164+0000 7fcbfd59c700 1 -- 172.21.15.135:0/1819956635 >> 172.21.15.135:0/1819956635 conn(0x7fcbf80f02e0 msgr2=0x7fcbf80f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:33.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.164+0000 7fcbfd59c700 1 -- 172.21.15.135:0/1819956635 shutdown_connections 2023-11-15T06:42:33.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.164+0000 7fcbfd59c700 1 -- 172.21.15.135:0/1819956635 wait complete. 2023-11-15T06:42:33.170 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbfd59c700 1 Processor -- start 2023-11-15T06:42:33.170 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbfd59c700 1 -- start start 2023-11-15T06:42:33.170 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbfd59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f7470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:33.170 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbfd59c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbf80f58f0 con 0x7fcbf80f4880 2023-11-15T06:42:33.170 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbf6ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f7470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:33.171 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbf6ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f7470 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:52032/0 (socket says 172.21.15.135:52032) 2023-11-15T06:42:33.171 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbf6ffd700 1 -- 172.21.15.135:0/2750768136 learned_addr learned my addr 172.21.15.135:0/2750768136 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:33.171 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbf6ffd700 1 -- 172.21.15.135:0/2750768136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbec005040 con 0x7fcbf80f4880 2023-11-15T06:42:33.172 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.165+0000 7fcbf6ffd700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f7470 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fcbec013510 tx=0x7fcbec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:33.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.166+0000 7fcbdf7fe700 1 -- 172.21.15.135:0/2750768136 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbec009320 con 0x7fcbf80f4880 2023-11-15T06:42:33.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.166+0000 7fcbdf7fe700 1 -- 172.21.15.135:0/2750768136 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbec013f00 con 0x7fcbf80f4880 2023-11-15T06:42:33.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.166+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbf80f5a90 con 0x7fcbf80f4880 2023-11-15T06:42:33.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.166+0000 7fcbdf7fe700 1 -- 172.21.15.135:0/2750768136 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbec014dc0 con 0x7fcbf80f4880 2023-11-15T06:42:33.173 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.166+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbf80f5f20 con 0x7fcbf80f4880 2023-11-15T06:42:33.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.167+0000 7fcbdf7fe700 1 -- 172.21.15.135:0/2750768136 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbec0095f0 con 0x7fcbf80f4880 2023-11-15T06:42:33.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.167+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbf8059100 con 0x7fcbf80f4880 2023-11-15T06:42:33.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.167+0000 7fcbdf7fe700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcbe002ff60 0x7fcbe0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:33.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.167+0000 7fcbdf7fe700 1 -- 172.21.15.135:0/2750768136 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbec041000 con 0x7fcbf80f4880 2023-11-15T06:42:33.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.167+0000 7fcbf67fc700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcbe002ff60 0x7fcbe0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:33.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.167+0000 7fcbf67fc700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcbe002ff60 0x7fcbe0032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fcbe80096a0 tx=0x7fcbe8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:33.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.171+0000 7fcbdf7fe700 1 -- 172.21.15.135:0/2750768136 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbec014420 con 0x7fcbf80f4880 2023-11-15T06:42:33.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.378+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbf80fbf50 con 0x7fcbf80f4880 2023-11-15T06:42:33.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.378+0000 7fcbdf7fe700 1 -- 172.21.15.135:0/2750768136 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcbec010230 con 0x7fcbf80f4880 2023-11-15T06:42:33.382 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:33.382 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:33.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcbe002ff60 msgr2=0x7fcbe0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:33.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcbe002ff60 0x7fcbe0032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7fcbe80096a0 tx=0x7fcbe8006b40).stop 2023-11-15T06:42:33.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 msgr2=0x7fcbf80f7470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:33.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f7470 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7fcbec013510 tx=0x7fcbec00b040).stop 2023-11-15T06:42:33.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 shutdown_connections 2023-11-15T06:42:33.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcbe002ff60 0x7fcbe0032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:33.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 --2- 172.21.15.135:0/2750768136 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcbf80f4880 0x7fcbf80f7470 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:33.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 >> 172.21.15.135:0/2750768136 conn(0x7fcbf80f02e0 msgr2=0x7fcbf80f0de0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:33.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 shutdown_connections 2023-11-15T06:42:33.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:33.380+0000 7fcbfd59c700 1 -- 172.21.15.135:0/2750768136 wait complete. 2023-11-15T06:42:33.386 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:33.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:32.276474+0000 mgr.smithi123.razjtn (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:33.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:33.379559+0000 mon.smithi123 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.135:0/2750768136' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:34.759 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:34.759 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:35.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:34.276823+0000 mgr.smithi123.razjtn (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.907+0000 7f8671fad700 1 -- 172.21.15.135:0/978113200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 msgr2=0x7f866c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:35.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.907+0000 7f8671fad700 1 --2- 172.21.15.135:0/978113200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c0f5830 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f8660009fb0 tx=0x7f8660009b30).stop 2023-11-15T06:42:35.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.907+0000 7f8671fad700 1 -- 172.21.15.135:0/978113200 shutdown_connections 2023-11-15T06:42:35.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.907+0000 7f8671fad700 1 --2- 172.21.15.135:0/978113200 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c0f5830 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:35.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.907+0000 7f8671fad700 1 -- 172.21.15.135:0/978113200 >> 172.21.15.135:0/978113200 conn(0x7f866c0f0b70 msgr2=0x7f866c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:35.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.907+0000 7f8671fad700 1 -- 172.21.15.135:0/978113200 shutdown_connections 2023-11-15T06:42:35.911 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.907+0000 7f8671fad700 1 -- 172.21.15.135:0/978113200 wait complete. 2023-11-15T06:42:35.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.908+0000 7f8671fad700 1 Processor -- start 2023-11-15T06:42:35.913 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.908+0000 7f8671fad700 1 -- start start 2023-11-15T06:42:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.908+0000 7f8671fad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.908+0000 7f8671fad700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f866c105b20 con 0x7f866c0f5410 2023-11-15T06:42:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.908+0000 7f866b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:35.914 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f866b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:52048/0 (socket says 172.21.15.135:52048) 2023-11-15T06:42:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f866b7fe700 1 -- 172.21.15.135:0/205263023 learned_addr learned my addr 172.21.15.135:0/205263023 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f866b7fe700 1 -- 172.21.15.135:0/205263023 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8660005040 con 0x7f866c0f5410 2023-11-15T06:42:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f866b7fe700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c105630 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f8660014000 tx=0x7f866000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:35.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f8653fff700 1 -- 172.21.15.135:0/205263023 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f86600095f0 con 0x7f866c0f5410 2023-11-15T06:42:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f866c105d20 con 0x7f866c0f5410 2023-11-15T06:42:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f8653fff700 1 -- 172.21.15.135:0/205263023 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8660007e70 con 0x7f866c0f5410 2023-11-15T06:42:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f8653fff700 1 -- 172.21.15.135:0/205263023 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8660014ea0 con 0x7f866c0f5410 2023-11-15T06:42:35.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.909+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f866c1061b0 con 0x7f866c0f5410 2023-11-15T06:42:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.910+0000 7f8653fff700 1 -- 172.21.15.135:0/205263023 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8660025070 con 0x7f866c0f5410 2023-11-15T06:42:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.910+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f866c045ad0 con 0x7f866c0f5410 2023-11-15T06:42:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.910+0000 7f8653fff700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f865402ff60 0x7f8654032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.910+0000 7f8653fff700 1 -- 172.21.15.135:0/205263023 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f86600409f0 con 0x7f866c0f5410 2023-11-15T06:42:35.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.910+0000 7f866affd700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f865402ff60 0x7f8654032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.911+0000 7f866affd700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f865402ff60 0x7f8654032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f865c0096a0 tx=0x7f865c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:35.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:35.915+0000 7f8653fff700 1 -- 172.21.15.135:0/205263023 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f866000f4e0 con 0x7f866c0f5410 2023-11-15T06:42:36.125 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.122+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f866c0faf50 con 0x7f866c0f5410 2023-11-15T06:42:36.126 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.123+0000 7f8653fff700 1 -- 172.21.15.135:0/205263023 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8660012bc0 con 0x7f866c0f5410 2023-11-15T06:42:36.127 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:36.127 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:36.128 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f865402ff60 msgr2=0x7f8654032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:36.128 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f865402ff60 0x7f8654032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f865c0096a0 tx=0x7f865c006b40).stop 2023-11-15T06:42:36.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 msgr2=0x7f866c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:36.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c105630 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f8660014000 tx=0x7f866000b040).stop 2023-11-15T06:42:36.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 shutdown_connections 2023-11-15T06:42:36.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f865402ff60 0x7f8654032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:36.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 --2- 172.21.15.135:0/205263023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f866c0f5410 0x7f866c105630 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:36.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 >> 172.21.15.135:0/205263023 conn(0x7f866c0f0b70 msgr2=0x7f866c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:36.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 shutdown_connections 2023-11-15T06:42:36.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:36.125+0000 7f8671fad700 1 -- 172.21.15.135:0/205263023 wait complete. 2023-11-15T06:42:36.130 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:36.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:36 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:36.124096+0000 mon.smithi123 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.135:0/205263023' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:37.503 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:37.504 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:37.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:36.277167+0000 mgr.smithi123.razjtn (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:38.621 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 -- 172.21.15.135:0/306604992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 msgr2=0x7f04300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:38.622 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 --2- 172.21.15.135:0/306604992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f04300f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f0418009fb0 tx=0x7f0418009b30).stop 2023-11-15T06:42:38.622 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 -- 172.21.15.135:0/306604992 shutdown_connections 2023-11-15T06:42:38.622 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 --2- 172.21.15.135:0/306604992 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f04300f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:38.623 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 -- 172.21.15.135:0/306604992 >> 172.21.15.135:0/306604992 conn(0x7f04300f0b70 msgr2=0x7f04300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:38.623 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 -- 172.21.15.135:0/306604992 shutdown_connections 2023-11-15T06:42:38.623 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 -- 172.21.15.135:0/306604992 wait complete. 2023-11-15T06:42:38.623 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 Processor -- start 2023-11-15T06:42:38.624 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.616+0000 7f0435095700 1 -- start start 2023-11-15T06:42:38.624 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.617+0000 7f0435095700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f0430109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:38.624 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.617+0000 7f0435095700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0430060240 con 0x7f04300f5410 2023-11-15T06:42:38.625 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.617+0000 7f042ed9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f0430109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:38.625 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.617+0000 7f042ed9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f0430109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:35284/0 (socket says 172.21.15.135:35284) 2023-11-15T06:42:38.625 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.617+0000 7f042ed9d700 1 -- 172.21.15.135:0/4222635078 learned_addr learned my addr 172.21.15.135:0/4222635078 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:38.625 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.617+0000 7f042ed9d700 1 -- 172.21.15.135:0/4222635078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0418005040 con 0x7f04300f5410 2023-11-15T06:42:38.626 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.617+0000 7f042ed9d700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f0430109430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f0418013510 tx=0x7f041800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:38.626 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f04277fe700 1 -- 172.21.15.135:0/4222635078 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0418009320 con 0x7f04300f5410 2023-11-15T06:42:38.626 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0430060440 con 0x7f04300f5410 2023-11-15T06:42:38.627 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f04277fe700 1 -- 172.21.15.135:0/4222635078 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0418013f00 con 0x7f04300f5410 2023-11-15T06:42:38.627 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f04277fe700 1 -- 172.21.15.135:0/4222635078 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0418014d90 con 0x7f04300f5410 2023-11-15T06:42:38.627 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0430109ae0 con 0x7f04300f5410 2023-11-15T06:42:38.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f04277fe700 1 -- 172.21.15.135:0/4222635078 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04180263a0 con 0x7f04300f5410 2023-11-15T06:42:38.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f04277fe700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f041c02ff60 0x7f041c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:38.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.618+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0430045ad0 con 0x7f04300f5410 2023-11-15T06:42:38.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.619+0000 7f04277fe700 1 -- 172.21.15.135:0/4222635078 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f04180410b0 con 0x7f04300f5410 2023-11-15T06:42:38.629 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.619+0000 7f042e59c700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f041c02ff60 0x7f041c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:38.630 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.619+0000 7f042e59c700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f041c02ff60 0x7f041c032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f04200096a0 tx=0x7f0420006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:38.630 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.623+0000 7f04277fe700 1 -- 172.21.15.135:0/4222635078 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f041806d2b0 con 0x7f04300f5410 2023-11-15T06:42:38.834 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.831+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f04300faf50 con 0x7f04300f5410 2023-11-15T06:42:38.835 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.832+0000 7f04277fe700 1 -- 172.21.15.135:0/4222635078 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f041801db70 con 0x7f04300f5410 2023-11-15T06:42:38.835 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:38.836 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:38.837 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f041c02ff60 msgr2=0x7f041c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:38.837 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f041c02ff60 0x7f041c032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f04200096a0 tx=0x7f0420006b40).stop 2023-11-15T06:42:38.837 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 msgr2=0x7f0430109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:38.837 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f0430109430 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f0418013510 tx=0x7f041800b040).stop 2023-11-15T06:42:38.837 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 shutdown_connections 2023-11-15T06:42:38.838 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f041c02ff60 0x7f041c032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:38.838 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 --2- 172.21.15.135:0/4222635078 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f04300f5410 0x7f0430109430 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:38.838 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 >> 172.21.15.135:0/4222635078 conn(0x7f04300f0b70 msgr2=0x7f04300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:38.838 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 shutdown_connections 2023-11-15T06:42:38.838 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:38.834+0000 7f0435095700 1 -- 172.21.15.135:0/4222635078 wait complete. 2023-11-15T06:42:38.839 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:39.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:38.277505+0000 mgr.smithi123.razjtn (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:39.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:38.833245+0000 mon.smithi123 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.135:0/4222635078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:40.167 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:40.168 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:40.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:40.367127+0000 mon.smithi123 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:42:40.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:40.373438+0000 mon.smithi123 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:42:41.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.298+0000 7fbc067d6700 1 -- 172.21.15.135:0/3388541015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 msgr2=0x7fbc000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:41.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.298+0000 7fbc067d6700 1 --2- 172.21.15.135:0/3388541015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc000f55c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7fbbf0009fb0 tx=0x7fbbf0009b30).stop 2023-11-15T06:42:41.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 -- 172.21.15.135:0/3388541015 shutdown_connections 2023-11-15T06:42:41.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 --2- 172.21.15.135:0/3388541015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc000f55c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:41.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 -- 172.21.15.135:0/3388541015 >> 172.21.15.135:0/3388541015 conn(0x7fbc000f09a0 msgr2=0x7fbc000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:41.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 -- 172.21.15.135:0/3388541015 shutdown_connections 2023-11-15T06:42:41.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 -- 172.21.15.135:0/3388541015 wait complete. 2023-11-15T06:42:41.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 Processor -- start 2023-11-15T06:42:41.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 -- start start 2023-11-15T06:42:41.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc001091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:41.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.299+0000 7fbc067d6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc00060240 con 0x7fbc000f51a0 2023-11-15T06:42:41.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbc057d4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc001091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:41.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbc057d4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc001091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:35306/0 (socket says 172.21.15.135:35306) 2023-11-15T06:42:41.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbc057d4700 1 -- 172.21.15.135:0/1786030822 learned_addr learned my addr 172.21.15.135:0/1786030822 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:41.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbc057d4700 1 -- 172.21.15.135:0/1786030822 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbbf0005040 con 0x7fbc000f51a0 2023-11-15T06:42:41.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbc057d4700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc001091b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fbbf0013510 tx=0x7fbbf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:41.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbbf5ffb700 1 -- 172.21.15.135:0/1786030822 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf0009320 con 0x7fbc000f51a0 2023-11-15T06:42:41.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc00060440 con 0x7fbc000f51a0 2023-11-15T06:42:41.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbbf5ffb700 1 -- 172.21.15.135:0/1786030822 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbbf0013f00 con 0x7fbc000f51a0 2023-11-15T06:42:41.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.300+0000 7fbbf5ffb700 1 -- 172.21.15.135:0/1786030822 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbbf0014dc0 con 0x7fbc000f51a0 2023-11-15T06:42:41.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.301+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc00109860 con 0x7fbc000f51a0 2023-11-15T06:42:41.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.301+0000 7fbbf5ffb700 1 -- 172.21.15.135:0/1786030822 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbbf001d3d0 con 0x7fbc000f51a0 2023-11-15T06:42:41.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.301+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc00045ad0 con 0x7fbc000f51a0 2023-11-15T06:42:41.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.301+0000 7fbbf5ffb700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbbec02ff60 0x7fbbec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:41.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.302+0000 7fbbf5ffb700 1 -- 172.21.15.135:0/1786030822 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbbf00413b0 con 0x7fbc000f51a0 2023-11-15T06:42:41.311 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.302+0000 7fbc04fd3700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbbec02ff60 0x7fbbec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:41.311 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.302+0000 7fbc04fd3700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbbec02ff60 0x7fbbec032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:41.311 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.306+0000 7fbbf5ffb700 1 -- 172.21.15.135:0/1786030822 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbbf00126f0 con 0x7fbc000f51a0 2023-11-15T06:42:41.517 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.514+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc0010a070 con 0x7fbc000f51a0 2023-11-15T06:42:41.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.515+0000 7fbbf5ffb700 1 -- 172.21.15.135:0/1786030822 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbbf0012330 con 0x7fbc000f51a0 2023-11-15T06:42:41.519 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:41.519 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:41.521 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbbec02ff60 msgr2=0x7fbbec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:41.521 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbbec02ff60 0x7fbbec032420 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7fbbfc0096a0 tx=0x7fbbfc006b40).stop 2023-11-15T06:42:41.521 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 msgr2=0x7fbc001091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:41.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc001091b0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7fbbf0013510 tx=0x7fbbf000b040).stop 2023-11-15T06:42:41.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 shutdown_connections 2023-11-15T06:42:41.522 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbbec02ff60 0x7fbbec032420 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:41.523 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 --2- 172.21.15.135:0/1786030822 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc000f51a0 0x7fbc001091b0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:41.523 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 >> 172.21.15.135:0/1786030822 conn(0x7fbc000f09a0 msgr2=0x7fbc000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:41.524 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 shutdown_connections 2023-11-15T06:42:41.524 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:41.517+0000 7fbc067d6700 1 -- 172.21.15.135:0/1786030822 wait complete. 2023-11-15T06:42:41.524 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:41.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:40.277883+0000 mgr.smithi123.razjtn (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:41.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:41.516296+0000 mon.smithi123 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.135:0/1786030822' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:42.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:42.907 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:43.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:42.278159+0000 mgr.smithi123.razjtn (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:44.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 -- 172.21.15.135:0/2392061994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 msgr2=0x7f95800f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:44.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 --2- 172.21.15.135:0/2392061994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95800f55c0 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f9570009fb0 tx=0x7f9570009b30).stop 2023-11-15T06:42:44.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 -- 172.21.15.135:0/2392061994 shutdown_connections 2023-11-15T06:42:44.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 --2- 172.21.15.135:0/2392061994 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95800f55c0 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:44.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 -- 172.21.15.135:0/2392061994 >> 172.21.15.135:0/2392061994 conn(0x7f95800f09a0 msgr2=0x7f95800f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:44.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 -- 172.21.15.135:0/2392061994 shutdown_connections 2023-11-15T06:42:44.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 -- 172.21.15.135:0/2392061994 wait complete. 2023-11-15T06:42:44.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 Processor -- start 2023-11-15T06:42:44.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 -- start start 2023-11-15T06:42:44.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95801064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:44.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.036+0000 7f9585546700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95801069d0 con 0x7f95800f51a0 2023-11-15T06:42:44.043 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.037+0000 7f957ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95801064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:44.043 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.037+0000 7f957ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95801064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:35326/0 (socket says 172.21.15.135:35326) 2023-11-15T06:42:44.043 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.037+0000 7f957ffff700 1 -- 172.21.15.135:0/3572818366 learned_addr learned my addr 172.21.15.135:0/3572818366 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:44.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.037+0000 7f957ffff700 1 -- 172.21.15.135:0/3572818366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9570005040 con 0x7f95800f51a0 2023-11-15T06:42:44.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.037+0000 7f957ffff700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95801064e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f9570009830 tx=0x7f957000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:44.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.037+0000 7f957cff9700 1 -- 172.21.15.135:0/3572818366 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9570009320 con 0x7f95800f51a0 2023-11-15T06:42:44.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.038+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9580106bd0 con 0x7f95800f51a0 2023-11-15T06:42:44.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.038+0000 7f957cff9700 1 -- 172.21.15.135:0/3572818366 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95700136c0 con 0x7f95800f51a0 2023-11-15T06:42:44.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.038+0000 7f957cff9700 1 -- 172.21.15.135:0/3572818366 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f957001ee90 con 0x7f95800f51a0 2023-11-15T06:42:44.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.038+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9580107060 con 0x7f95800f51a0 2023-11-15T06:42:44.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.038+0000 7f957cff9700 1 -- 172.21.15.135:0/3572818366 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9570014440 con 0x7f95800f51a0 2023-11-15T06:42:44.045 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.038+0000 7f957cff9700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f956802ff10 0x7f95680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:44.046 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.039+0000 7f957cff9700 1 -- 172.21.15.135:0/3572818366 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9570040ed0 con 0x7f95800f51a0 2023-11-15T06:42:44.046 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.039+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9580045ad0 con 0x7f95800f51a0 2023-11-15T06:42:44.046 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.039+0000 7f957f7fe700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f956802ff10 0x7f95680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:44.046 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.039+0000 7f957f7fe700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f956802ff10 0x7f95680323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f95740096a0 tx=0x7f9574006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:44.047 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.043+0000 7f957cff9700 1 -- 172.21.15.135:0/3572818366 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f957001c910 con 0x7f95800f51a0 2023-11-15T06:42:44.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.252+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9580107d40 con 0x7f95800f51a0 2023-11-15T06:42:44.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.252+0000 7f957cff9700 1 -- 172.21.15.135:0/3572818366 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f957001c320 con 0x7f95800f51a0 2023-11-15T06:42:44.256 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:44.256 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:44.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f956802ff10 msgr2=0x7f95680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:44.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f956802ff10 0x7f95680323d0 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f95740096a0 tx=0x7f9574006b40).stop 2023-11-15T06:42:44.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 msgr2=0x7f95801064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:44.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95801064e0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f9570009830 tx=0x7f957000b040).stop 2023-11-15T06:42:44.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 shutdown_connections 2023-11-15T06:42:44.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f956802ff10 0x7f95680323d0 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:44.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 --2- 172.21.15.135:0/3572818366 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95800f51a0 0x7f95801064e0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:44.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 >> 172.21.15.135:0/3572818366 conn(0x7f95800f09a0 msgr2=0x7f958005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:44.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 shutdown_connections 2023-11-15T06:42:44.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:44.254+0000 7f9585546700 1 -- 172.21.15.135:0/3572818366 wait complete. 2023-11-15T06:42:44.259 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:44.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:44 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:44.253540+0000 mon.smithi123 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.135:0/3572818366' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:45.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:45.632 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:45.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:44.278517+0000 mgr.smithi123.razjtn (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:46.760 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.756+0000 7fb1e347b700 1 -- 172.21.15.135:0/1314267171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 msgr2=0x7fb1dc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:46.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.756+0000 7fb1e347b700 1 --2- 172.21.15.135:0/1314267171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc0f5830 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7fb1cc009fb0 tx=0x7fb1cc009b30).stop 2023-11-15T06:42:46.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.756+0000 7fb1e347b700 1 -- 172.21.15.135:0/1314267171 shutdown_connections 2023-11-15T06:42:46.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.756+0000 7fb1e347b700 1 --2- 172.21.15.135:0/1314267171 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc0f5830 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:46.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.756+0000 7fb1e347b700 1 -- 172.21.15.135:0/1314267171 >> 172.21.15.135:0/1314267171 conn(0x7fb1dc0f0b70 msgr2=0x7fb1dc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:46.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.756+0000 7fb1e347b700 1 -- 172.21.15.135:0/1314267171 shutdown_connections 2023-11-15T06:42:46.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.756+0000 7fb1e347b700 1 -- 172.21.15.135:0/1314267171 wait complete. 2023-11-15T06:42:46.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e347b700 1 Processor -- start 2023-11-15T06:42:46.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e347b700 1 -- start start 2023-11-15T06:42:46.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e347b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc1066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:46.763 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e347b700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb1dc106bc0 con 0x7fb1dc0f5410 2023-11-15T06:42:46.763 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e1217700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc1066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:46.763 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e1217700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc1066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:35346/0 (socket says 172.21.15.135:35346) 2023-11-15T06:42:46.763 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e1217700 1 -- 172.21.15.135:0/4090038470 learned_addr learned my addr 172.21.15.135:0/4090038470 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:46.763 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e1217700 1 -- 172.21.15.135:0/4090038470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb1cc005040 con 0x7fb1dc0f5410 2023-11-15T06:42:46.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.757+0000 7fb1e1217700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc1066d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fb1cc013510 tx=0x7fb1cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:46.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.758+0000 7fb1d1ffb700 1 -- 172.21.15.135:0/4090038470 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc009320 con 0x7fb1dc0f5410 2023-11-15T06:42:46.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.758+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1dc106dc0 con 0x7fb1dc0f5410 2023-11-15T06:42:46.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.758+0000 7fb1d1ffb700 1 -- 172.21.15.135:0/4090038470 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb1cc013f00 con 0x7fb1dc0f5410 2023-11-15T06:42:46.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.758+0000 7fb1d1ffb700 1 -- 172.21.15.135:0/4090038470 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb1cc014d90 con 0x7fb1dc0f5410 2023-11-15T06:42:46.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.758+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb1dc107250 con 0x7fb1dc0f5410 2023-11-15T06:42:46.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.759+0000 7fb1d1ffb700 1 -- 172.21.15.135:0/4090038470 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1cc014420 con 0x7fb1dc0f5410 2023-11-15T06:42:46.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.759+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb1dc045ad0 con 0x7fb1dc0f5410 2023-11-15T06:42:46.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.759+0000 7fb1d1ffb700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb1c802ffb0 0x7fb1c8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:46.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.759+0000 7fb1d1ffb700 1 -- 172.21.15.135:0/4090038470 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1cc0410b0 con 0x7fb1dc0f5410 2023-11-15T06:42:46.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.759+0000 7fb1e0a16700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb1c802ffb0 0x7fb1c8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:46.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.760+0000 7fb1e0a16700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb1c802ffb0 0x7fb1c8032470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb1d80096a0 tx=0x7fb1d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:46.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.764+0000 7fb1d1ffb700 1 -- 172.21.15.135:0/4090038470 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb1cc06d2b0 con 0x7fb1dc0f5410 2023-11-15T06:42:46.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.973+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1dc0fae60 con 0x7fb1dc0f5410 2023-11-15T06:42:46.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.973+0000 7fb1d1ffb700 1 -- 172.21.15.135:0/4090038470 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb1cc0095f0 con 0x7fb1dc0f5410 2023-11-15T06:42:46.976 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:46.977 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:46.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb1c802ffb0 msgr2=0x7fb1c8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:46.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb1c802ffb0 0x7fb1c8032470 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fb1d80096a0 tx=0x7fb1d8006b40).stop 2023-11-15T06:42:46.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 msgr2=0x7fb1dc1066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:46.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc1066d0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7fb1cc013510 tx=0x7fb1cc00b040).stop 2023-11-15T06:42:46.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 shutdown_connections 2023-11-15T06:42:46.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb1c802ffb0 0x7fb1c8032470 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:46.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 --2- 172.21.15.135:0/4090038470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb1dc0f5410 0x7fb1dc1066d0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:46.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 >> 172.21.15.135:0/4090038470 conn(0x7fb1dc0f0b70 msgr2=0x7fb1dc05fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:46.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 shutdown_connections 2023-11-15T06:42:46.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:46.975+0000 7fb1e347b700 1 -- 172.21.15.135:0/4090038470 wait complete. 2023-11-15T06:42:46.980 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:47.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:46.278815+0000 mgr.smithi123.razjtn (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:47.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:46.974444+0000 mon.smithi123 (mon.0) 227 : audit [DBG] from='client.? 172.21.15.135:0/4090038470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:48.363 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:48.364 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:48.897 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:48 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:48.553823+0000 mon.smithi123 (mon.0) 228 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2023-11-15T06:42:48.897 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:48 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:48.556361+0000 mon.smithi123 (mon.0) 229 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi123", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2023-11-15T06:42:48.898 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:48 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:48.557157+0000 mon.smithi123 (mon.0) 230 : audit [DBG] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2023-11-15T06:42:49.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 -- 172.21.15.135:0/4099921089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 msgr2=0x7f601c0f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:49.457 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 --2- 172.21.15.135:0/4099921089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c0f8f20 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f6018009fb0 tx=0x7f6018009b30).stop 2023-11-15T06:42:49.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 -- 172.21.15.135:0/4099921089 shutdown_connections 2023-11-15T06:42:49.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 --2- 172.21.15.135:0/4099921089 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c0f8f20 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:49.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 -- 172.21.15.135:0/4099921089 >> 172.21.15.135:0/4099921089 conn(0x7f601c0f09a0 msgr2=0x7f601c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:49.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 -- 172.21.15.135:0/4099921089 shutdown_connections 2023-11-15T06:42:49.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 -- 172.21.15.135:0/4099921089 wait complete. 2023-11-15T06:42:49.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 Processor -- start 2023-11-15T06:42:49.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 -- start start 2023-11-15T06:42:49.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:49.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.453+0000 7f6022a69700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f601c1069f0 con 0x7f601c0601e0 2023-11-15T06:42:49.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f6021a67700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:49.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f6021a67700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58638/0 (socket says 172.21.15.135:58638) 2023-11-15T06:42:49.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f6021a67700 1 -- 172.21.15.135:0/2493745057 learned_addr learned my addr 172.21.15.135:0/2493745057 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:49.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f6021a67700 1 -- 172.21.15.135:0/2493745057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6018005040 con 0x7f601c0601e0 2023-11-15T06:42:49.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f6021a67700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c106500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6018013510 tx=0x7f601800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:49.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f60127fc700 1 -- 172.21.15.135:0/2493745057 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6018009320 con 0x7f601c0601e0 2023-11-15T06:42:49.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f601c106bf0 con 0x7f601c0601e0 2023-11-15T06:42:49.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f60127fc700 1 -- 172.21.15.135:0/2493745057 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6018013f00 con 0x7f601c0601e0 2023-11-15T06:42:49.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f60127fc700 1 -- 172.21.15.135:0/2493745057 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6018014d90 con 0x7f601c0601e0 2023-11-15T06:42:49.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.454+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f601c107080 con 0x7f601c0601e0 2023-11-15T06:42:49.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.455+0000 7f60127fc700 1 -- 172.21.15.135:0/2493745057 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6018014420 con 0x7f601c0601e0 2023-11-15T06:42:49.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.455+0000 7f60127fc700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f600802ff60 0x7f6008032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:49.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.455+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f601c045ad0 con 0x7f601c0601e0 2023-11-15T06:42:49.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.455+0000 7f60127fc700 1 -- 172.21.15.135:0/2493745057 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60180410c0 con 0x7f601c0601e0 2023-11-15T06:42:49.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.455+0000 7f6021266700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f600802ff60 0x7f6008032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:49.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.456+0000 7f6021266700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f600802ff60 0x7f6008032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f600c0096a0 tx=0x7f600c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:49.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.460+0000 7f60127fc700 1 -- 172.21.15.135:0/2493745057 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60180095f0 con 0x7f601c0601e0 2023-11-15T06:42:49.669 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.667+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f601c107f30 con 0x7f601c0601e0 2023-11-15T06:42:49.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.667+0000 7f60127fc700 1 -- 172.21.15.135:0/2493745057 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f601801db70 con 0x7f601c0601e0 2023-11-15T06:42:49.671 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:49.671 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:49.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f600802ff60 msgr2=0x7f6008032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:49.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f600802ff60 0x7f6008032420 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f600c0096a0 tx=0x7f600c006b40).stop 2023-11-15T06:42:49.673 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 msgr2=0x7f601c106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:49.673 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c106500 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f6018013510 tx=0x7f601800b040).stop 2023-11-15T06:42:49.673 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 shutdown_connections 2023-11-15T06:42:49.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f600802ff60 0x7f6008032420 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:49.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 --2- 172.21.15.135:0/2493745057 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f601c0601e0 0x7f601c106500 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:49.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 >> 172.21.15.135:0/2493745057 conn(0x7f601c0f09a0 msgr2=0x7f601c0f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:49.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 shutdown_connections 2023-11-15T06:42:49.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:49.669+0000 7f6022a69700 1 -- 172.21.15.135:0/2493745057 wait complete. 2023-11-15T06:42:49.675 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:49.903 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:48.279143+0000 mgr.smithi123.razjtn (mgr.14162) 155 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:49.903 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:42:48.552489+0000 mgr.smithi123.razjtn (mgr.14162) 156 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi123 ... 2023-11-15T06:42:49.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T06:42:49.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:42:49.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:38:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.904 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:39:04Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:39:42Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:42:49.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T06:42:49.905 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:42:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:40:56Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:41:33Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:42:11Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:42:49.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Traceback (most recent call last): 2023-11-15T06:42:49.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:42:49.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: main() 2023-11-15T06:42:49.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:42:49.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: r = ctx.func(ctx) 2023-11-15T06:42:49.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:42:49.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: return func(ctx) 2023-11-15T06:42:49.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T06:42:49.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T06:42:49.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T06:42:49.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T06:42:49.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T06:42:49.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: raise RuntimeError('uid/gid not found') 2023-11-15T06:42:49.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: RuntimeError: uid/gid not found 2023-11-15T06:42:49.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Traceback (most recent call last): 2023-11-15T06:42:49.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T06:42:49.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: yield (conn, connr) 2023-11-15T06:42:49.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T06:42:49.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: code, '\n'.join(err))) 2023-11-15T06:42:49.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi123 ... 2023-11-15T06:42:49.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2023-11-15T06:42:49.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:42:49.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:38:27Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: copying system image from manifest list: determining manifest MIME type for docker://prom/alertmanager:v0.20.0: reading manifest sha256:b9323917a2eda265bec69e59a457f001c529facbbc8166df277f4850cdac61a0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:39:04Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:39:42Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:42:49.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi123 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2023-11-15T06:42:49.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2023-11-15T06:42:49.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:40:56Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:41:33Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr time="2023-11-15T06:42:11Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2023-11-15T06:42:49.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2023-11-15T06:42:49.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Traceback (most recent call last): 2023-11-15T06:42:49.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:42:49.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: main() 2023-11-15T06:42:49.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:42:49.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: r = ctx.func(ctx) 2023-11-15T06:42:49.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:42:49.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: return func(ctx) 2023-11-15T06:42:49.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2023-11-15T06:42:49.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2023-11-15T06:42:49.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2023-11-15T06:42:49.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2023-11-15T06:42:49.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2023-11-15T06:42:49.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: raise RuntimeError('uid/gid not found') 2023-11-15T06:42:49.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: RuntimeError: uid/gid not found 2023-11-15T06:42:49.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:42:48.557757+0000 mgr.smithi123.razjtn (mgr.14162) 157 : cephadm [INF] Deploying daemon crash.smithi123 on smithi123 2023-11-15T06:42:50.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:49.668428+0000 mon.smithi123 (mon.0) 231 : audit [DBG] from='client.? 172.21.15.135:0/2493745057' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:50.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:50.055937+0000 mon.smithi123 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:42:50.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:50.145591+0000 mon.smithi123 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:42:50.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:50.147638+0000 mon.smithi123 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:42:50.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:50.148045+0000 mon.smithi123 (mon.0) 235 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T06:42:50.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:50.148463+0000 mgr.smithi123.razjtn (mgr.14162) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2023-11-15T06:42:50.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:50.150668+0000 mon.smithi123 (mon.0) 236 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:42:50.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:42:50.153559+0000 mgr.smithi123.razjtn (mgr.14162) 159 : cephadm [INF] Deploying daemon grafana.smithi123 on smithi123 2023-11-15T06:42:51.053 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:51.054 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:52.159 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.155+0000 7f61bd9f7700 1 -- 172.21.15.135:0/765662023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 msgr2=0x7f61b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:52.159 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.155+0000 7f61bd9f7700 1 --2- 172.21.15.135:0/765662023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b80f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f61a8009fb0 tx=0x7f61a8009b30).stop 2023-11-15T06:42:52.159 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.155+0000 7f61bd9f7700 1 -- 172.21.15.135:0/765662023 shutdown_connections 2023-11-15T06:42:52.160 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.155+0000 7f61bd9f7700 1 --2- 172.21.15.135:0/765662023 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b80f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:52.160 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.155+0000 7f61bd9f7700 1 -- 172.21.15.135:0/765662023 >> 172.21.15.135:0/765662023 conn(0x7f61b80f1c70 msgr2=0x7f61b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:52.160 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.155+0000 7f61bd9f7700 1 -- 172.21.15.135:0/765662023 shutdown_connections 2023-11-15T06:42:52.160 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.155+0000 7f61bd9f7700 1 -- 172.21.15.135:0/765662023 wait complete. 2023-11-15T06:42:52.160 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.156+0000 7f61bd9f7700 1 Processor -- start 2023-11-15T06:42:52.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.156+0000 7f61bd9f7700 1 -- start start 2023-11-15T06:42:52.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.156+0000 7f61bd9f7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:52.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.156+0000 7f61bd9f7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61b8106c20 con 0x7f61b80f6510 2023-11-15T06:42:52.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f61b6ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:52.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f61b6ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58656/0 (socket says 172.21.15.135:58656) 2023-11-15T06:42:52.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f61b6ffd700 1 -- 172.21.15.135:0/2250099030 learned_addr learned my addr 172.21.15.135:0/2250099030 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:52.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f61b6ffd700 1 -- 172.21.15.135:0/2250099030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f61a8005040 con 0x7f61b80f6510 2023-11-15T06:42:52.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f61b6ffd700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b8106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f61a8013510 tx=0x7f61a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:52.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f619f7fe700 1 -- 172.21.15.135:0/2250099030 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61a8009320 con 0x7f61b80f6510 2023-11-15T06:42:52.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f61b8106e20 con 0x7f61b80f6510 2023-11-15T06:42:52.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f619f7fe700 1 -- 172.21.15.135:0/2250099030 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61a8013f00 con 0x7f61b80f6510 2023-11-15T06:42:52.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f619f7fe700 1 -- 172.21.15.135:0/2250099030 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f61a8014d90 con 0x7f61b80f6510 2023-11-15T06:42:52.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.157+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f61b81072b0 con 0x7f61b80f6510 2023-11-15T06:42:52.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.158+0000 7f619f7fe700 1 -- 172.21.15.135:0/2250099030 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f61a8014420 con 0x7f61b80f6510 2023-11-15T06:42:52.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.158+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f61b8045ad0 con 0x7f61b80f6510 2023-11-15T06:42:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.158+0000 7f619f7fe700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f61a002ff60 0x7f61a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.158+0000 7f619f7fe700 1 -- 172.21.15.135:0/2250099030 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f61a8041000 con 0x7f61b80f6510 2023-11-15T06:42:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.159+0000 7f61b67fc700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f61a002ff60 0x7f61a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:52.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.159+0000 7f61b67fc700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f61a002ff60 0x7f61a0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f61ac0096a0 tx=0x7f61ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:52.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.163+0000 7f619f7fe700 1 -- 172.21.15.135:0/2250099030 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f61a80095f0 con 0x7f61b80f6510 2023-11-15T06:42:52.373 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.371+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f61b80fbf60 con 0x7f61b80f6510 2023-11-15T06:42:52.374 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.371+0000 7f619f7fe700 1 -- 172.21.15.135:0/2250099030 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f61a801db70 con 0x7f61b80f6510 2023-11-15T06:42:52.375 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:52.375 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:52.376 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f61a002ff60 msgr2=0x7f61a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:52.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f61a002ff60 0x7f61a0032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f61ac0096a0 tx=0x7f61ac006b40).stop 2023-11-15T06:42:52.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 msgr2=0x7f61b8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:52.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b8106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f61a8013510 tx=0x7f61a800b040).stop 2023-11-15T06:42:52.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 shutdown_connections 2023-11-15T06:42:52.378 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f61a002ff60 0x7f61a0032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:52.378 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 --2- 172.21.15.135:0/2250099030 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61b80f6510 0x7f61b8106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:52.378 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 >> 172.21.15.135:0/2250099030 conn(0x7f61b80f1c70 msgr2=0x7f61b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:52.378 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 shutdown_connections 2023-11-15T06:42:52.378 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:52.373+0000 7f61bd9f7700 1 -- 172.21.15.135:0/2250099030 wait complete. 2023-11-15T06:42:52.379 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:52.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:52 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:50.279394+0000 mgr.smithi123.razjtn (mgr.14162) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:53.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:52.372419+0000 mon.smithi123 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.135:0/2250099030' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:53.748 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:53.749 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:54.406 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:54 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:52.279683+0000 mgr.smithi123.razjtn (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:54.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.912+0000 7f15cacdb700 1 -- 172.21.15.135:0/2994989708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 msgr2=0x7f15c40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:54.915 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.912+0000 7f15cacdb700 1 --2- 172.21.15.135:0/2994989708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c40f6930 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7f15b8009fb0 tx=0x7f15b8009b30).stop 2023-11-15T06:42:54.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.912+0000 7f15cacdb700 1 -- 172.21.15.135:0/2994989708 shutdown_connections 2023-11-15T06:42:54.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.912+0000 7f15cacdb700 1 --2- 172.21.15.135:0/2994989708 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c40f6930 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:54.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.912+0000 7f15cacdb700 1 -- 172.21.15.135:0/2994989708 >> 172.21.15.135:0/2994989708 conn(0x7f15c40f1c70 msgr2=0x7f15c40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:54.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.912+0000 7f15cacdb700 1 -- 172.21.15.135:0/2994989708 shutdown_connections 2023-11-15T06:42:54.916 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.912+0000 7f15cacdb700 1 -- 172.21.15.135:0/2994989708 wait complete. 2023-11-15T06:42:54.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.913+0000 7f15cacdb700 1 Processor -- start 2023-11-15T06:42:54.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.913+0000 7f15cacdb700 1 -- start start 2023-11-15T06:42:54.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.913+0000 7f15cacdb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:54.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.913+0000 7f15cacdb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15c4106c20 con 0x7f15c40f6510 2023-11-15T06:42:54.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.913+0000 7f15c8a77700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:54.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.913+0000 7f15c8a77700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58670/0 (socket says 172.21.15.135:58670) 2023-11-15T06:42:54.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.913+0000 7f15c8a77700 1 -- 172.21.15.135:0/2600039326 learned_addr learned my addr 172.21.15.135:0/2600039326 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:54.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.914+0000 7f15c8a77700 1 -- 172.21.15.135:0/2600039326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15b8005040 con 0x7f15c40f6510 2023-11-15T06:42:54.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.914+0000 7f15c8a77700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c4106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f15b8013510 tx=0x7f15b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:54.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.914+0000 7f15c17fa700 1 -- 172.21.15.135:0/2600039326 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15b8009320 con 0x7f15c40f6510 2023-11-15T06:42:54.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.914+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15c4106e20 con 0x7f15c40f6510 2023-11-15T06:42:54.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.914+0000 7f15c17fa700 1 -- 172.21.15.135:0/2600039326 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15b8013f00 con 0x7f15c40f6510 2023-11-15T06:42:54.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.914+0000 7f15c17fa700 1 -- 172.21.15.135:0/2600039326 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15b8014d90 con 0x7f15c40f6510 2023-11-15T06:42:54.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.914+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15c41072b0 con 0x7f15c40f6510 2023-11-15T06:42:54.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.915+0000 7f15c17fa700 1 -- 172.21.15.135:0/2600039326 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15b80095f0 con 0x7f15c40f6510 2023-11-15T06:42:54.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.915+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15c4045ad0 con 0x7f15c40f6510 2023-11-15T06:42:54.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.915+0000 7f15c17fa700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f15b402ff60 0x7f15b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:54.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.915+0000 7f15c17fa700 1 -- 172.21.15.135:0/2600039326 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15b8040fe0 con 0x7f15c40f6510 2023-11-15T06:42:54.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.915+0000 7f15c3fff700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f15b402ff60 0x7f15b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:54.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.915+0000 7f15c3fff700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f15b402ff60 0x7f15b4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f15b00096a0 tx=0x7f15b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:54.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:54.920+0000 7f15c17fa700 1 -- 172.21.15.135:0/2600039326 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15b801d7f0 con 0x7f15c40f6510 2023-11-15T06:42:55.128 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.125+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15c40fbf60 con 0x7f15c40f6510 2023-11-15T06:42:55.129 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.126+0000 7f15c17fa700 1 -- 172.21.15.135:0/2600039326 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f15b8010230 con 0x7f15c40f6510 2023-11-15T06:42:55.129 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:55.130 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:55.130 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f15b402ff60 msgr2=0x7f15b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:55.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f15b402ff60 0x7f15b4032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7f15b00096a0 tx=0x7f15b0006b40).stop 2023-11-15T06:42:55.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 msgr2=0x7f15c4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:55.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c4106730 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7f15b8013510 tx=0x7f15b800b040).stop 2023-11-15T06:42:55.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 shutdown_connections 2023-11-15T06:42:55.131 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f15b402ff60 0x7f15b4032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:55.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 --2- 172.21.15.135:0/2600039326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f15c40f6510 0x7f15c4106730 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:55.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 >> 172.21.15.135:0/2600039326 conn(0x7f15c40f1c70 msgr2=0x7f15c40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:55.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 shutdown_connections 2023-11-15T06:42:55.132 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:55.128+0000 7f15cacdb700 1 -- 172.21.15.135:0/2600039326 wait complete. 2023-11-15T06:42:55.133 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:55.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:55.127127+0000 mon.smithi123 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.135:0/2600039326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:56.517 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:56.518 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:42:56.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:56 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:54.280022+0000 mgr.smithi123.razjtn (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:57.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.577+0000 7f260fb11700 1 -- 172.21.15.135:0/205229406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 msgr2=0x7f26080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:57.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.577+0000 7f260fb11700 1 --2- 172.21.15.135:0/205229406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f26080f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f25f8009fb0 tx=0x7f25f8009b30).stop 2023-11-15T06:42:57.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.577+0000 7f260fb11700 1 -- 172.21.15.135:0/205229406 shutdown_connections 2023-11-15T06:42:57.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.577+0000 7f260fb11700 1 --2- 172.21.15.135:0/205229406 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f26080f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:57.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.577+0000 7f260fb11700 1 -- 172.21.15.135:0/205229406 >> 172.21.15.135:0/205229406 conn(0x7f26080f0b70 msgr2=0x7f26080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:57.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.577+0000 7f260fb11700 1 -- 172.21.15.135:0/205229406 shutdown_connections 2023-11-15T06:42:57.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.577+0000 7f260fb11700 1 -- 172.21.15.135:0/205229406 wait complete. 2023-11-15T06:42:57.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260fb11700 1 Processor -- start 2023-11-15T06:42:57.583 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260fb11700 1 -- start start 2023-11-15T06:42:57.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260fb11700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f2608106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:57.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260fb11700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2608106c10 con 0x7f26080f5410 2023-11-15T06:42:57.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260d8ad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f2608106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:57.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260d8ad700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f2608106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:58678/0 (socket says 172.21.15.135:58678) 2023-11-15T06:42:57.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260d8ad700 1 -- 172.21.15.135:0/1383145659 learned_addr learned my addr 172.21.15.135:0/1383145659 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:42:57.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260d8ad700 1 -- 172.21.15.135:0/1383145659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f25f8005040 con 0x7f26080f5410 2023-11-15T06:42:57.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.578+0000 7f260d8ad700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f2608106720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f25f8013510 tx=0x7f25f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:57.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.579+0000 7f25fe7fc700 1 -- 172.21.15.135:0/1383145659 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f8009320 con 0x7f26080f5410 2023-11-15T06:42:57.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.579+0000 7f25fe7fc700 1 -- 172.21.15.135:0/1383145659 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f25f8013f00 con 0x7f26080f5410 2023-11-15T06:42:57.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.579+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2608106e10 con 0x7f26080f5410 2023-11-15T06:42:57.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.579+0000 7f25fe7fc700 1 -- 172.21.15.135:0/1383145659 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f25f8014d90 con 0x7f26080f5410 2023-11-15T06:42:57.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.579+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f26081072a0 con 0x7f26080f5410 2023-11-15T06:42:57.587 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.579+0000 7f25fe7fc700 1 -- 172.21.15.135:0/1383145659 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25f8014420 con 0x7f26080f5410 2023-11-15T06:42:57.587 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.579+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2608045ad0 con 0x7f26080f5410 2023-11-15T06:42:57.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.580+0000 7f25fe7fc700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f402ff60 0x7f25f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:42:57.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.580+0000 7f25fe7fc700 1 -- 172.21.15.135:0/1383145659 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25f80410c0 con 0x7f26080f5410 2023-11-15T06:42:57.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.580+0000 7f260d0ac700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f402ff60 0x7f25f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:42:57.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.580+0000 7f260d0ac700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f402ff60 0x7f25f4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f26040096a0 tx=0x7f2604006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:42:57.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.584+0000 7f25fe7fc700 1 -- 172.21.15.135:0/1383145659 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f25f80095f0 con 0x7f26080f5410 2023-11-15T06:42:57.793 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.791+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2608108150 con 0x7f26080f5410 2023-11-15T06:42:57.794 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.791+0000 7f25fe7fc700 1 -- 172.21.15.135:0/1383145659 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f25f801db70 con 0x7f26080f5410 2023-11-15T06:42:57.794 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:42:57.795 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:42:57.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f402ff60 msgr2=0x7f25f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:57.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f402ff60 0x7f25f4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f26040096a0 tx=0x7f2604006b40).stop 2023-11-15T06:42:57.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 msgr2=0x7f2608106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:42:57.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f2608106720 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f25f8013510 tx=0x7f25f800b040).stop 2023-11-15T06:42:57.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 shutdown_connections 2023-11-15T06:42:57.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f25f402ff60 0x7f25f4032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:57.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 --2- 172.21.15.135:0/1383145659 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f26080f5410 0x7f2608106720 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:42:57.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 >> 172.21.15.135:0/1383145659 conn(0x7f26080f0b70 msgr2=0x7f260805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:42:57.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 shutdown_connections 2023-11-15T06:42:57.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:42:57.793+0000 7f260fb11700 1 -- 172.21.15.135:0/1383145659 wait complete. 2023-11-15T06:42:57.798 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:42:58.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:56.280366+0000 mgr.smithi123.razjtn (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:42:58.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:42:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:42:57.792359+0000 mon.smithi123 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.135:0/1383145659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:42:59.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:42:59.177 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:00.364 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3797208102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 msgr2=0x7fdbfc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:00.364 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 --2- 172.21.15.135:0/3797208102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fdbf0009fb0 tx=0x7fdbf0009b30).stop 2023-11-15T06:43:00.365 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3797208102 shutdown_connections 2023-11-15T06:43:00.365 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 --2- 172.21.15.135:0/3797208102 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:00.365 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3797208102 >> 172.21.15.135:0/3797208102 conn(0x7fdbfc0f1c70 msgr2=0x7fdbfc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:00.365 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3797208102 shutdown_connections 2023-11-15T06:43:00.365 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3797208102 wait complete. 2023-11-15T06:43:00.366 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.361+0000 7fdc01f8d700 1 Processor -- start 2023-11-15T06:43:00.367 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdc01f8d700 1 -- start start 2023-11-15T06:43:00.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdc01f8d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:00.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdc01f8d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdbfc106c20 con 0x7fdbfc0f6510 2023-11-15T06:43:00.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdbfb7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:00.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdbfb7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:45310/0 (socket says 172.21.15.135:45310) 2023-11-15T06:43:00.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdbfb7fe700 1 -- 172.21.15.135:0/3955858051 learned_addr learned my addr 172.21.15.135:0/3955858051 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:00.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdbfb7fe700 1 -- 172.21.15.135:0/3955858051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdbf0005040 con 0x7fdbfc0f6510 2023-11-15T06:43:00.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.362+0000 7fdbfb7fe700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fdbf0013510 tx=0x7fdbf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:00.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.363+0000 7fdbe3fff700 1 -- 172.21.15.135:0/3955858051 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf0009320 con 0x7fdbfc0f6510 2023-11-15T06:43:00.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.363+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdbfc106e20 con 0x7fdbfc0f6510 2023-11-15T06:43:00.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.363+0000 7fdbe3fff700 1 -- 172.21.15.135:0/3955858051 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdbf0013f00 con 0x7fdbfc0f6510 2023-11-15T06:43:00.370 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.363+0000 7fdbe3fff700 1 -- 172.21.15.135:0/3955858051 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdbf0014d90 con 0x7fdbfc0f6510 2023-11-15T06:43:00.370 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.363+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdbfc1072b0 con 0x7fdbfc0f6510 2023-11-15T06:43:00.370 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.363+0000 7fdbe3fff700 1 -- 172.21.15.135:0/3955858051 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdbf00263a0 con 0x7fdbfc0f6510 2023-11-15T06:43:00.370 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.364+0000 7fdbe3fff700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdbe402ff60 0x7fdbe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:00.370 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.364+0000 7fdbe3fff700 1 -- 172.21.15.135:0/3955858051 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdbf00412d0 con 0x7fdbfc0f6510 2023-11-15T06:43:00.371 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.364+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdbfc059100 con 0x7fdbfc0f6510 2023-11-15T06:43:00.371 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.364+0000 7fdbfaffd700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdbe402ff60 0x7fdbe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:00.371 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.364+0000 7fdbfaffd700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdbe402ff60 0x7fdbe4032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fdbec0096a0 tx=0x7fdbec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:00.371 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.368+0000 7fdbe3fff700 1 -- 172.21.15.135:0/3955858051 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdbf001cb10 con 0x7fdbfc0f6510 2023-11-15T06:43:00.575 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.572+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdbfc0fbf60 con 0x7fdbfc0f6510 2023-11-15T06:43:00.576 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.573+0000 7fdbe3fff700 1 -- 172.21.15.135:0/3955858051 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdbf002b030 con 0x7fdbfc0f6510 2023-11-15T06:43:00.577 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:00.577 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:00.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdbe402ff60 msgr2=0x7fdbe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:00.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdbe402ff60 0x7fdbe4032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fdbec0096a0 tx=0x7fdbec006b40).stop 2023-11-15T06:43:00.579 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 msgr2=0x7fdbfc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:00.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc106730 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fdbf0013510 tx=0x7fdbf000b040).stop 2023-11-15T06:43:00.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 shutdown_connections 2023-11-15T06:43:00.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdbe402ff60 0x7fdbe4032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:00.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 --2- 172.21.15.135:0/3955858051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdbfc0f6510 0x7fdbfc106730 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:00.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 >> 172.21.15.135:0/3955858051 conn(0x7fdbfc0f1c70 msgr2=0x7fdbfc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:00.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 shutdown_connections 2023-11-15T06:43:00.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:00.574+0000 7fdc01f8d700 1 -- 172.21.15.135:0/3955858051 wait complete. 2023-11-15T06:43:00.581 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:00.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:00 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:42:58.280603+0000 mgr.smithi123.razjtn (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:01.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:00.573945+0000 mon.smithi123 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.135:0/3955858051' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:01.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:01.939 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:02.656 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:02 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:00.280758+0000 mgr.smithi123.razjtn (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:03.050 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.045+0000 7f5747fff700 1 -- 172.21.15.135:0/2902477202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 msgr2=0x7f57480f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:03.050 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.045+0000 7f5747fff700 1 --2- 172.21.15.135:0/2902477202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57480f55c0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f5738009fb0 tx=0x7f5738009b30).stop 2023-11-15T06:43:03.050 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.045+0000 7f5747fff700 1 -- 172.21.15.135:0/2902477202 shutdown_connections 2023-11-15T06:43:03.050 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.045+0000 7f5747fff700 1 --2- 172.21.15.135:0/2902477202 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57480f55c0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:03.051 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.045+0000 7f5747fff700 1 -- 172.21.15.135:0/2902477202 >> 172.21.15.135:0/2902477202 conn(0x7f57480f09a0 msgr2=0x7f57480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:03.051 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.045+0000 7f5747fff700 1 -- 172.21.15.135:0/2902477202 shutdown_connections 2023-11-15T06:43:03.051 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.045+0000 7f5747fff700 1 -- 172.21.15.135:0/2902477202 wait complete. 2023-11-15T06:43:03.051 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.046+0000 7f5747fff700 1 Processor -- start 2023-11-15T06:43:03.051 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.046+0000 7f5747fff700 1 -- start start 2023-11-15T06:43:03.052 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.046+0000 7f5747fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57481091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:03.052 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.046+0000 7f5747fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5748060240 con 0x7f57480f51a0 2023-11-15T06:43:03.052 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.046+0000 7f5746ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57481091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:03.052 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.046+0000 7f5746ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57481091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:45336/0 (socket says 172.21.15.135:45336) 2023-11-15T06:43:03.052 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.046+0000 7f5746ffd700 1 -- 172.21.15.135:0/673632712 learned_addr learned my addr 172.21.15.135:0/673632712 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:03.053 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.047+0000 7f5746ffd700 1 -- 172.21.15.135:0/673632712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5738005040 con 0x7f57480f51a0 2023-11-15T06:43:03.053 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.047+0000 7f5746ffd700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57481091b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5738013510 tx=0x7f573800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:03.053 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.047+0000 7f572f7fe700 1 -- 172.21.15.135:0/673632712 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5738009320 con 0x7f57480f51a0 2023-11-15T06:43:03.053 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.047+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5748060440 con 0x7f57480f51a0 2023-11-15T06:43:03.054 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.047+0000 7f572f7fe700 1 -- 172.21.15.135:0/673632712 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5738013f00 con 0x7f57480f51a0 2023-11-15T06:43:03.054 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.047+0000 7f572f7fe700 1 -- 172.21.15.135:0/673632712 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5738014dc0 con 0x7f57480f51a0 2023-11-15T06:43:03.054 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.047+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5748109860 con 0x7f57480f51a0 2023-11-15T06:43:03.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.048+0000 7f572f7fe700 1 -- 172.21.15.135:0/673632712 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5738014420 con 0x7f57480f51a0 2023-11-15T06:43:03.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.048+0000 7f572f7fe700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f573002ff60 0x7f5730032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:03.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.048+0000 7f572f7fe700 1 -- 172.21.15.135:0/673632712 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57380410d0 con 0x7f57480f51a0 2023-11-15T06:43:03.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.048+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5748045ad0 con 0x7f57480f51a0 2023-11-15T06:43:03.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.048+0000 7f57467fc700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f573002ff60 0x7f5730032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:03.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.048+0000 7f57467fc700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f573002ff60 0x7f5730032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:03.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.052+0000 7f572f7fe700 1 -- 172.21.15.135:0/673632712 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57380095f0 con 0x7f57480f51a0 2023-11-15T06:43:03.263 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.261+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5748109fe0 con 0x7f57480f51a0 2023-11-15T06:43:03.264 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.261+0000 7f572f7fe700 1 -- 172.21.15.135:0/673632712 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f573802b020 con 0x7f57480f51a0 2023-11-15T06:43:03.265 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:03.265 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:03.266 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f573002ff60 msgr2=0x7f5730032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:03.266 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f573002ff60 0x7f5730032420 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f573c0096a0 tx=0x7f573c006b40).stop 2023-11-15T06:43:03.266 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 msgr2=0x7f57481091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:03.266 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57481091b0 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f5738013510 tx=0x7f573800b040).stop 2023-11-15T06:43:03.266 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 shutdown_connections 2023-11-15T06:43:03.267 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f573002ff60 0x7f5730032420 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:03.267 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 --2- 172.21.15.135:0/673632712 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f57480f51a0 0x7f57481091b0 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:03.267 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 >> 172.21.15.135:0/673632712 conn(0x7f57480f09a0 msgr2=0x7f57480f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:03.267 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 shutdown_connections 2023-11-15T06:43:03.267 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:03.263+0000 7f5747fff700 1 -- 172.21.15.135:0/673632712 wait complete. 2023-11-15T06:43:03.268 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:04.495 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:04 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:02.280961+0000 mgr.smithi123.razjtn (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:04.496 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:04 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:03.262527+0000 mon.smithi123 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.135:0/673632712' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:04.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:04.604 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:05.723 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.719+0000 7fbc7f78c700 1 -- 172.21.15.135:0/2506991302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 msgr2=0x7fbc780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:05.724 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.719+0000 7fbc7f78c700 1 --2- 172.21.15.135:0/2506991302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc780f6930 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7fbc70009960 tx=0x7fbc70004d10).stop 2023-11-15T06:43:05.724 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.719+0000 7fbc7f78c700 1 -- 172.21.15.135:0/2506991302 shutdown_connections 2023-11-15T06:43:05.724 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.719+0000 7fbc7f78c700 1 --2- 172.21.15.135:0/2506991302 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc780f6930 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:05.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.719+0000 7fbc7f78c700 1 -- 172.21.15.135:0/2506991302 >> 172.21.15.135:0/2506991302 conn(0x7fbc780f1c70 msgr2=0x7fbc780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:05.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.719+0000 7fbc7f78c700 1 -- 172.21.15.135:0/2506991302 shutdown_connections 2023-11-15T06:43:05.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.719+0000 7fbc7f78c700 1 -- 172.21.15.135:0/2506991302 wait complete. 2023-11-15T06:43:05.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7f78c700 1 Processor -- start 2023-11-15T06:43:05.725 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7f78c700 1 -- start start 2023-11-15T06:43:05.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7f78c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc78106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:05.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7f78c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc78106c20 con 0x7fbc780f6510 2023-11-15T06:43:05.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7d528700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc78106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:05.726 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7d528700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc78106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:45340/0 (socket says 172.21.15.135:45340) 2023-11-15T06:43:05.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7d528700 1 -- 172.21.15.135:0/426530185 learned_addr learned my addr 172.21.15.135:0/426530185 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:05.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7d528700 1 -- 172.21.15.135:0/426530185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc70009640 con 0x7fbc780f6510 2023-11-15T06:43:05.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7d528700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc78106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fbc70004b70 tx=0x7fbc70004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:05.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc75ffb700 1 -- 172.21.15.135:0/426530185 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc70018070 con 0x7fbc780f6510 2023-11-15T06:43:05.727 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc78106e20 con 0x7fbc780f6510 2023-11-15T06:43:05.728 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc75ffb700 1 -- 172.21.15.135:0/426530185 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc7000b870 con 0x7fbc780f6510 2023-11-15T06:43:05.728 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.720+0000 7fbc75ffb700 1 -- 172.21.15.135:0/426530185 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc7001c560 con 0x7fbc780f6510 2023-11-15T06:43:05.728 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.721+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc781072b0 con 0x7fbc780f6510 2023-11-15T06:43:05.729 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.721+0000 7fbc75ffb700 1 -- 172.21.15.135:0/426530185 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc70013630 con 0x7fbc780f6510 2023-11-15T06:43:05.729 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.721+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc78045ad0 con 0x7fbc780f6510 2023-11-15T06:43:05.729 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.721+0000 7fbc75ffb700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbc6002ff60 0x7fbc60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:05.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.722+0000 7fbc75ffb700 1 -- 172.21.15.135:0/426530185 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc700413f0 con 0x7fbc780f6510 2023-11-15T06:43:05.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.722+0000 7fbc7cd27700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbc6002ff60 0x7fbc60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:05.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.722+0000 7fbc7cd27700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbc6002ff60 0x7fbc60032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fbc6c0096a0 tx=0x7fbc6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:05.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.726+0000 7fbc75ffb700 1 -- 172.21.15.135:0/426530185 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc70004810 con 0x7fbc780f6510 2023-11-15T06:43:05.936 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.933+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc780fbf60 con 0x7fbc780f6510 2023-11-15T06:43:05.937 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.933+0000 7fbc75ffb700 1 -- 172.21.15.135:0/426530185 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc70012380 con 0x7fbc780f6510 2023-11-15T06:43:05.937 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:05.937 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:05.938 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbc6002ff60 msgr2=0x7fbc60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:05.939 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbc6002ff60 0x7fbc60032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7fbc6c0096a0 tx=0x7fbc6c006b40).stop 2023-11-15T06:43:05.939 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 msgr2=0x7fbc78106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:05.939 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc78106730 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7fbc70004b70 tx=0x7fbc70004d10).stop 2023-11-15T06:43:05.939 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 shutdown_connections 2023-11-15T06:43:05.940 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbc6002ff60 0x7fbc60032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:05.940 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 --2- 172.21.15.135:0/426530185 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbc780f6510 0x7fbc78106730 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:05.940 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 >> 172.21.15.135:0/426530185 conn(0x7fbc780f1c70 msgr2=0x7fbc780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:05.940 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 shutdown_connections 2023-11-15T06:43:05.941 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:05.935+0000 7fbc7f78c700 1 -- 172.21.15.135:0/426530185 wait complete. 2023-11-15T06:43:05.941 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:06.422 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:06 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:04.281215+0000 mgr.smithi123.razjtn (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:06.422 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:06 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:05.934725+0000 mon.smithi123 (mon.0) 242 : audit [DBG] from='client.? 172.21.15.135:0/426530185' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:07.313 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:07.314 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:07.657 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:06.281463+0000 mgr.smithi123.razjtn (mgr.14162) 168 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:07.657 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:06.306485+0000 mon.smithi123 (mon.0) 243 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' 2023-11-15T06:43:07.657 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:43:06.307800+0000 mgr.smithi123.razjtn (mgr.14162) 169 : cephadm [INF] Deploying daemon node-exporter.smithi123 on smithi123 2023-11-15T06:43:08.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.423+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/1747656131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 msgr2=0x7f2f140f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:08.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.423+0000 7f2f1c6c7700 1 --2- 172.21.15.135:0/1747656131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f140f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7f2f0c009fb0 tx=0x7f2f0c009b30).stop 2023-11-15T06:43:08.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.423+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/1747656131 shutdown_connections 2023-11-15T06:43:08.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.423+0000 7f2f1c6c7700 1 --2- 172.21.15.135:0/1747656131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f140f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:08.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.423+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/1747656131 >> 172.21.15.135:0/1747656131 conn(0x7f2f140f1c70 msgr2=0x7f2f140f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:08.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.423+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/1747656131 shutdown_connections 2023-11-15T06:43:08.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.423+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/1747656131 wait complete. 2023-11-15T06:43:08.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1c6c7700 1 Processor -- start 2023-11-15T06:43:08.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1c6c7700 1 -- start start 2023-11-15T06:43:08.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1c6c7700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f14106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:08.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1c6c7700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f14106c20 con 0x7f2f140f6510 2023-11-15T06:43:08.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1a463700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f14106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:08.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1a463700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f14106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48856/0 (socket says 172.21.15.135:48856) 2023-11-15T06:43:08.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1a463700 1 -- 172.21.15.135:0/894960084 learned_addr learned my addr 172.21.15.135:0/894960084 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:08.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1a463700 1 -- 172.21.15.135:0/894960084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f0c005040 con 0x7f2f140f6510 2023-11-15T06:43:08.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.424+0000 7f2f1a463700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f14106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2f0c013510 tx=0x7f2f0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:08.433 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.425+0000 7f2f12ffd700 1 -- 172.21.15.135:0/894960084 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0c009320 con 0x7f2f140f6510 2023-11-15T06:43:08.433 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.425+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f14106e20 con 0x7f2f140f6510 2023-11-15T06:43:08.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.425+0000 7f2f12ffd700 1 -- 172.21.15.135:0/894960084 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f0c013f00 con 0x7f2f140f6510 2023-11-15T06:43:08.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.425+0000 7f2f12ffd700 1 -- 172.21.15.135:0/894960084 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f0c014d90 con 0x7f2f140f6510 2023-11-15T06:43:08.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.425+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f141072b0 con 0x7f2f140f6510 2023-11-15T06:43:08.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.425+0000 7f2f12ffd700 1 -- 172.21.15.135:0/894960084 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f0c014420 con 0x7f2f140f6510 2023-11-15T06:43:08.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.425+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f14045ad0 con 0x7f2f140f6510 2023-11-15T06:43:08.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.426+0000 7f2f12ffd700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2efc02ff60 0x7f2efc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:08.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.426+0000 7f2f12ffd700 1 -- 172.21.15.135:0/894960084 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f0c0410c0 con 0x7f2f140f6510 2023-11-15T06:43:08.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.426+0000 7f2f19c62700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2efc02ff60 0x7f2efc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:08.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.426+0000 7f2f19c62700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2efc02ff60 0x7f2efc032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f2f080096a0 tx=0x7f2f08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:08.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.431+0000 7f2f12ffd700 1 -- 172.21.15.135:0/894960084 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f0c0095f0 con 0x7f2f140f6510 2023-11-15T06:43:08.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.635+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f140fbf60 con 0x7f2f140f6510 2023-11-15T06:43:08.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.636+0000 7f2f12ffd700 1 -- 172.21.15.135:0/894960084 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2f0c01db70 con 0x7f2f140f6510 2023-11-15T06:43:08.641 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:08.641 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:08.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2efc02ff60 msgr2=0x7f2efc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:08.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2efc02ff60 0x7f2efc032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f2f080096a0 tx=0x7f2f08006b40).stop 2023-11-15T06:43:08.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 msgr2=0x7f2f14106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:08.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f14106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7f2f0c013510 tx=0x7f2f0c00b040).stop 2023-11-15T06:43:08.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 shutdown_connections 2023-11-15T06:43:08.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2efc02ff60 0x7f2efc032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:08.644 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 --2- 172.21.15.135:0/894960084 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2f140f6510 0x7f2f14106730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:08.644 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 >> 172.21.15.135:0/894960084 conn(0x7f2f140f1c70 msgr2=0x7f2f140f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:08.644 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 shutdown_connections 2023-11-15T06:43:08.644 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:08.638+0000 7f2f1c6c7700 1 -- 172.21.15.135:0/894960084 wait complete. 2023-11-15T06:43:08.645 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:08.281692+0000 mgr.smithi123.razjtn (mgr.14162) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:09.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:08.638432+0000 mon.smithi123 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.135:0/894960084' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:10.022 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:10.023 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:11.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.202+0000 7fdc9f14a700 1 -- 172.21.15.135:0/3246812415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 msgr2=0x7fdc980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:11.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.202+0000 7fdc9f14a700 1 --2- 172.21.15.135:0/3246812415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc980f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7fdc8c009fb0 tx=0x7fdc8c009b30).stop 2023-11-15T06:43:11.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.202+0000 7fdc9f14a700 1 -- 172.21.15.135:0/3246812415 shutdown_connections 2023-11-15T06:43:11.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.202+0000 7fdc9f14a700 1 --2- 172.21.15.135:0/3246812415 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc980f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:11.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.202+0000 7fdc9f14a700 1 -- 172.21.15.135:0/3246812415 >> 172.21.15.135:0/3246812415 conn(0x7fdc980f0b70 msgr2=0x7fdc980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:11.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.202+0000 7fdc9f14a700 1 -- 172.21.15.135:0/3246812415 shutdown_connections 2023-11-15T06:43:11.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.202+0000 7fdc9f14a700 1 -- 172.21.15.135:0/3246812415 wait complete. 2023-11-15T06:43:11.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.203+0000 7fdc9f14a700 1 Processor -- start 2023-11-15T06:43:11.213 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.203+0000 7fdc9f14a700 1 -- start start 2023-11-15T06:43:11.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.203+0000 7fdc9f14a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc98109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:11.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.203+0000 7fdc9f14a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdc98060240 con 0x7fdc980f5410 2023-11-15T06:43:11.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.203+0000 7fdc9cee6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc98109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:11.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.203+0000 7fdc9cee6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc98109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48870/0 (socket says 172.21.15.135:48870) 2023-11-15T06:43:11.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.203+0000 7fdc9cee6700 1 -- 172.21.15.135:0/2978249964 learned_addr learned my addr 172.21.15.135:0/2978249964 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:11.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.204+0000 7fdc9cee6700 1 -- 172.21.15.135:0/2978249964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdc8c005040 con 0x7fdc980f5410 2023-11-15T06:43:11.217 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.204+0000 7fdc9cee6700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc98109430 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fdc8c014000 tx=0x7fdc8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:11.217 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.204+0000 7fdc957fa700 1 -- 172.21.15.135:0/2978249964 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc8c0095f0 con 0x7fdc980f5410 2023-11-15T06:43:11.217 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.204+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdc98060440 con 0x7fdc980f5410 2023-11-15T06:43:11.218 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.204+0000 7fdc957fa700 1 -- 172.21.15.135:0/2978249964 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdc8c007e70 con 0x7fdc980f5410 2023-11-15T06:43:11.218 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.204+0000 7fdc957fa700 1 -- 172.21.15.135:0/2978249964 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdc8c014ea0 con 0x7fdc980f5410 2023-11-15T06:43:11.218 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.204+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdc98109ae0 con 0x7fdc980f5410 2023-11-15T06:43:11.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.205+0000 7fdc957fa700 1 -- 172.21.15.135:0/2978249964 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdc8c025070 con 0x7fdc980f5410 2023-11-15T06:43:11.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.205+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdc98045ad0 con 0x7fdc980f5410 2023-11-15T06:43:11.219 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.205+0000 7fdc957fa700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdc8802ff60 0x7fdc88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:11.220 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.205+0000 7fdc957fa700 1 -- 172.21.15.135:0/2978249964 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdc8c040a30 con 0x7fdc980f5410 2023-11-15T06:43:11.220 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.205+0000 7fdc97fff700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdc8802ff60 0x7fdc88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:11.220 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.205+0000 7fdc97fff700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdc8802ff60 0x7fdc88032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fdc840096a0 tx=0x7fdc84006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:11.221 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.209+0000 7fdc957fa700 1 -- 172.21.15.135:0/2978249964 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdc8c0079c0 con 0x7fdc980f5410 2023-11-15T06:43:11.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.414+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdc980fae60 con 0x7fdc980f5410 2023-11-15T06:43:11.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.415+0000 7fdc957fa700 1 -- 172.21.15.135:0/2978249964 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdc8c00f300 con 0x7fdc980f5410 2023-11-15T06:43:11.420 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:11.420 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:11.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdc8802ff60 msgr2=0x7fdc88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:11.422 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdc8802ff60 0x7fdc88032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fdc840096a0 tx=0x7fdc84006b40).stop 2023-11-15T06:43:11.422 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 msgr2=0x7fdc98109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:11.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc98109430 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7fdc8c014000 tx=0x7fdc8c00b040).stop 2023-11-15T06:43:11.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 shutdown_connections 2023-11-15T06:43:11.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fdc8802ff60 0x7fdc88032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:11.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 --2- 172.21.15.135:0/2978249964 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fdc980f5410 0x7fdc98109430 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:11.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.416+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 >> 172.21.15.135:0/2978249964 conn(0x7fdc980f0b70 msgr2=0x7fdc980f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:11.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.417+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 shutdown_connections 2023-11-15T06:43:11.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:11.417+0000 7fdc9f14a700 1 -- 172.21.15.135:0/2978249964 wait complete. 2023-11-15T06:43:11.425 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:11.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:10.282047+0000 mgr.smithi123.razjtn (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:11.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:11.416878+0000 mon.smithi123 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.135:0/2978249964' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:12.812 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:12.812 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:13.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:12.282326+0000 mgr.smithi123.razjtn (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:13.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.984+0000 7ff001a21700 1 -- 172.21.15.135:0/218770564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 msgr2=0x7feffc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:13.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.984+0000 7ff001a21700 1 --2- 172.21.15.135:0/218770564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc0f5830 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7feff0009fb0 tx=0x7feff0009b30).stop 2023-11-15T06:43:13.988 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.984+0000 7ff001a21700 1 -- 172.21.15.135:0/218770564 shutdown_connections 2023-11-15T06:43:13.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.984+0000 7ff001a21700 1 --2- 172.21.15.135:0/218770564 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc0f5830 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:13.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.984+0000 7ff001a21700 1 -- 172.21.15.135:0/218770564 >> 172.21.15.135:0/218770564 conn(0x7feffc0f0b70 msgr2=0x7feffc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:13.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.984+0000 7ff001a21700 1 -- 172.21.15.135:0/218770564 shutdown_connections 2023-11-15T06:43:13.989 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.984+0000 7ff001a21700 1 -- 172.21.15.135:0/218770564 wait complete. 2023-11-15T06:43:13.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7ff001a21700 1 Processor -- start 2023-11-15T06:43:13.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7ff001a21700 1 -- start start 2023-11-15T06:43:13.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7ff001a21700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:13.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7ff001a21700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feffc105b20 con 0x7feffc0f5410 2023-11-15T06:43:13.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7feffaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:13.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7feffaffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48876/0 (socket says 172.21.15.135:48876) 2023-11-15T06:43:13.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7feffaffd700 1 -- 172.21.15.135:0/2022834935 learned_addr learned my addr 172.21.15.135:0/2022834935 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:13.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.985+0000 7feffaffd700 1 -- 172.21.15.135:0/2022834935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feff0005040 con 0x7feffc0f5410 2023-11-15T06:43:13.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.986+0000 7feffaffd700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc105630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7feff0013510 tx=0x7feff000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:13.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.986+0000 7fefe37fe700 1 -- 172.21.15.135:0/2022834935 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feff0009320 con 0x7feffc0f5410 2023-11-15T06:43:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.986+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feffc105d20 con 0x7feffc0f5410 2023-11-15T06:43:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.986+0000 7fefe37fe700 1 -- 172.21.15.135:0/2022834935 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feff0013f00 con 0x7feffc0f5410 2023-11-15T06:43:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.986+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feffc1061b0 con 0x7feffc0f5410 2023-11-15T06:43:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.986+0000 7fefe37fe700 1 -- 172.21.15.135:0/2022834935 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feff0014d90 con 0x7feffc0f5410 2023-11-15T06:43:13.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.986+0000 7fefe37fe700 1 -- 172.21.15.135:0/2022834935 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feff0014420 con 0x7feffc0f5410 2023-11-15T06:43:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.987+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feffc045ad0 con 0x7feffc0f5410 2023-11-15T06:43:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.987+0000 7fefe37fe700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fefe402ff60 0x7fefe4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.987+0000 7fefe37fe700 1 -- 172.21.15.135:0/2022834935 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feff0041040 con 0x7feffc0f5410 2023-11-15T06:43:13.995 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.987+0000 7feffa7fc700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fefe402ff60 0x7fefe4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:13.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.987+0000 7feffa7fc700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fefe402ff60 0x7fefe4032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fefec0096a0 tx=0x7fefec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:13.996 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:13.992+0000 7fefe37fe700 1 -- 172.21.15.135:0/2022834935 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feff00095f0 con 0x7feffc0f5410 2023-11-15T06:43:14.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.199+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feffc0fae60 con 0x7feffc0f5410 2023-11-15T06:43:14.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7fefe37fe700 1 -- 172.21.15.135:0/2022834935 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feff001db70 con 0x7feffc0f5410 2023-11-15T06:43:14.204 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:14.204 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:14.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fefe402ff60 msgr2=0x7fefe4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:14.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fefe402ff60 0x7fefe4032420 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fefec0096a0 tx=0x7fefec006b40).stop 2023-11-15T06:43:14.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 msgr2=0x7feffc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:14.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc105630 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7feff0013510 tx=0x7feff000b040).stop 2023-11-15T06:43:14.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 shutdown_connections 2023-11-15T06:43:14.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fefe402ff60 0x7fefe4032420 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:14.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 --2- 172.21.15.135:0/2022834935 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7feffc0f5410 0x7feffc105630 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:14.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.201+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 >> 172.21.15.135:0/2022834935 conn(0x7feffc0f0b70 msgr2=0x7feffc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:14.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.202+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 shutdown_connections 2023-11-15T06:43:14.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:14.202+0000 7ff001a21700 1 -- 172.21.15.135:0/2022834935 wait complete. 2023-11-15T06:43:14.208 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:14.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:14 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:14.202002+0000 mon.smithi123 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.135:0/2022834935' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:15.586 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:15.587 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:15.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:14.282668+0000 mgr.smithi123.razjtn (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:16.760 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 -- 172.21.15.135:0/2951265163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 msgr2=0x7f1c6c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:16.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 --2- 172.21.15.135:0/2951265163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c0f6930 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7f1c5c009fb0 tx=0x7f1c5c009b30).stop 2023-11-15T06:43:16.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 -- 172.21.15.135:0/2951265163 shutdown_connections 2023-11-15T06:43:16.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 --2- 172.21.15.135:0/2951265163 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c0f6930 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:16.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 -- 172.21.15.135:0/2951265163 >> 172.21.15.135:0/2951265163 conn(0x7f1c6c0f1c70 msgr2=0x7f1c6c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:16.761 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 -- 172.21.15.135:0/2951265163 shutdown_connections 2023-11-15T06:43:16.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 -- 172.21.15.135:0/2951265163 wait complete. 2023-11-15T06:43:16.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.756+0000 7f1c7235c700 1 Processor -- start 2023-11-15T06:43:16.762 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.757+0000 7f1c7235c700 1 -- start start 2023-11-15T06:43:16.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.757+0000 7f1c7235c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:16.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.757+0000 7f1c7235c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c6c106c20 con 0x7f1c6c0f6510 2023-11-15T06:43:16.764 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.757+0000 7f1c6bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:16.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.757+0000 7f1c6bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48894/0 (socket says 172.21.15.135:48894) 2023-11-15T06:43:16.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.757+0000 7f1c6bfff700 1 -- 172.21.15.135:0/2849173239 learned_addr learned my addr 172.21.15.135:0/2849173239 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:16.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.757+0000 7f1c6bfff700 1 -- 172.21.15.135:0/2849173239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1c5c005040 con 0x7f1c6c0f6510 2023-11-15T06:43:16.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.758+0000 7f1c6bfff700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f1c5c013510 tx=0x7f1c5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:16.765 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.758+0000 7f1c68ff9700 1 -- 172.21.15.135:0/2849173239 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c009320 con 0x7f1c6c0f6510 2023-11-15T06:43:16.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.758+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c6c106e20 con 0x7f1c6c0f6510 2023-11-15T06:43:16.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.758+0000 7f1c68ff9700 1 -- 172.21.15.135:0/2849173239 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1c5c013f00 con 0x7f1c6c0f6510 2023-11-15T06:43:16.766 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.758+0000 7f1c68ff9700 1 -- 172.21.15.135:0/2849173239 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1c5c014d90 con 0x7f1c6c0f6510 2023-11-15T06:43:16.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.758+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c6c1072b0 con 0x7f1c6c0f6510 2023-11-15T06:43:16.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.759+0000 7f1c68ff9700 1 -- 172.21.15.135:0/2849173239 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1c5c014420 con 0x7f1c6c0f6510 2023-11-15T06:43:16.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.759+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c6c045ad0 con 0x7f1c6c0f6510 2023-11-15T06:43:16.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.759+0000 7f1c68ff9700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1c5802ff60 0x7f1c58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:16.767 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.759+0000 7f1c68ff9700 1 -- 172.21.15.135:0/2849173239 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1c5c040fb0 con 0x7f1c6c0f6510 2023-11-15T06:43:16.768 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.759+0000 7f1c6b7fe700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1c5802ff60 0x7f1c58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:16.768 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.759+0000 7f1c6b7fe700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1c5802ff60 0x7f1c58032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1c540096a0 tx=0x7f1c54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:16.768 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.764+0000 7f1c68ff9700 1 -- 172.21.15.135:0/2849173239 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1c5c0095f0 con 0x7f1c6c0f6510 2023-11-15T06:43:16.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.967+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c6c0fbf60 con 0x7f1c6c0f6510 2023-11-15T06:43:16.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.968+0000 7f1c68ff9700 1 -- 172.21.15.135:0/2849173239 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1c5c01db70 con 0x7f1c6c0f6510 2023-11-15T06:43:16.972 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:16.973 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:16.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1c5802ff60 msgr2=0x7f1c58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:16.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1c5802ff60 0x7f1c58032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7f1c540096a0 tx=0x7f1c54006b40).stop 2023-11-15T06:43:16.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 msgr2=0x7f1c6c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:16.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c106730 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7f1c5c013510 tx=0x7f1c5c00b040).stop 2023-11-15T06:43:16.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 shutdown_connections 2023-11-15T06:43:16.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1c5802ff60 0x7f1c58032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:16.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 --2- 172.21.15.135:0/2849173239 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c6c0f6510 0x7f1c6c106730 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:16.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 >> 172.21.15.135:0/2849173239 conn(0x7f1c6c0f1c70 msgr2=0x7f1c6c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:16.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 shutdown_connections 2023-11-15T06:43:16.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:16.970+0000 7f1c7235c700 1 -- 172.21.15.135:0/2849173239 wait complete. 2023-11-15T06:43:16.976 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:17.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:16.283006+0000 mgr.smithi123.razjtn (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:17.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:16.970169+0000 mon.smithi123 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.135:0/2849173239' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:18.362 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:18.363 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:19.481 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 -- 172.21.15.135:0/1223441963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 msgr2=0x7f79e80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:19.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 --2- 172.21.15.135:0/1223441963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e80f55c0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f79dc009fb0 tx=0x7f79dc009b30).stop 2023-11-15T06:43:19.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 -- 172.21.15.135:0/1223441963 shutdown_connections 2023-11-15T06:43:19.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 --2- 172.21.15.135:0/1223441963 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e80f55c0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:19.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 -- 172.21.15.135:0/1223441963 >> 172.21.15.135:0/1223441963 conn(0x7f79e80f09a0 msgr2=0x7f79e80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:19.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 -- 172.21.15.135:0/1223441963 shutdown_connections 2023-11-15T06:43:19.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 -- 172.21.15.135:0/1223441963 wait complete. 2023-11-15T06:43:19.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 Processor -- start 2023-11-15T06:43:19.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 -- start start 2023-11-15T06:43:19.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e81091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:19.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.477+0000 7f79edced700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f79e8060220 con 0x7f79e80f51a0 2023-11-15T06:43:19.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79ecceb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e81091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:19.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79ecceb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e81091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57766/0 (socket says 172.21.15.135:57766) 2023-11-15T06:43:19.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79ecceb700 1 -- 172.21.15.135:0/2899750748 learned_addr learned my addr 172.21.15.135:0/2899750748 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:19.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79ecceb700 1 -- 172.21.15.135:0/2899750748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f79dc005040 con 0x7f79e80f51a0 2023-11-15T06:43:19.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79ecceb700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e81091d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f79dc013510 tx=0x7f79dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:19.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79e57fa700 1 -- 172.21.15.135:0/2899750748 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79dc009320 con 0x7f79e80f51a0 2023-11-15T06:43:19.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f79e8060420 con 0x7f79e80f51a0 2023-11-15T06:43:19.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79e57fa700 1 -- 172.21.15.135:0/2899750748 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f79dc013f00 con 0x7f79e80f51a0 2023-11-15T06:43:19.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79e57fa700 1 -- 172.21.15.135:0/2899750748 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f79dc014dc0 con 0x7f79e80f51a0 2023-11-15T06:43:19.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.478+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f79e8109880 con 0x7f79e80f51a0 2023-11-15T06:43:19.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.479+0000 7f79e57fa700 1 -- 172.21.15.135:0/2899750748 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f79dc014420 con 0x7f79e80f51a0 2023-11-15T06:43:19.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.479+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f79e8059100 con 0x7f79e80f51a0 2023-11-15T06:43:19.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.479+0000 7f79e57fa700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f79d802ffb0 0x7f79d8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:19.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.479+0000 7f79e57fa700 1 -- 172.21.15.135:0/2899750748 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f79dc040fe0 con 0x7f79e80f51a0 2023-11-15T06:43:19.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.479+0000 7f79e7fff700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f79d802ffb0 0x7f79d8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:19.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.480+0000 7f79e7fff700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f79d802ffb0 0x7f79d8032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f79d40096a0 tx=0x7f79d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:19.489 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.484+0000 7f79e57fa700 1 -- 172.21.15.135:0/2899750748 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f79dc06d2b0 con 0x7f79e80f51a0 2023-11-15T06:43:19.694 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.690+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f79e8045ad0 con 0x7f79e80f51a0 2023-11-15T06:43:19.695 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.691+0000 7f79e57fa700 1 -- 172.21.15.135:0/2899750748 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f79dc0095f0 con 0x7f79e80f51a0 2023-11-15T06:43:19.695 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:19.696 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:19.696 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f79d802ffb0 msgr2=0x7f79d8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:19.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f79d802ffb0 0x7f79d8032470 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f79d40096a0 tx=0x7f79d4006b40).stop 2023-11-15T06:43:19.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 msgr2=0x7f79e81091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:19.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e81091d0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f79dc013510 tx=0x7f79dc00b040).stop 2023-11-15T06:43:19.697 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 shutdown_connections 2023-11-15T06:43:19.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f79d802ffb0 0x7f79d8032470 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:19.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 --2- 172.21.15.135:0/2899750748 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f79e80f51a0 0x7f79e81091d0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:19.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 >> 172.21.15.135:0/2899750748 conn(0x7f79e80f09a0 msgr2=0x7f79e80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:19.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 shutdown_connections 2023-11-15T06:43:19.698 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:19.693+0000 7f79edced700 1 -- 172.21.15.135:0/2899750748 wait complete. 2023-11-15T06:43:19.699 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:19.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:18.283358+0000 mgr.smithi123.razjtn (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:20.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:20 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:19.693299+0000 mon.smithi123 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.135:0/2899750748' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:21.079 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:21.080 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:21.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:20.283623+0000 mgr.smithi123.razjtn (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:22.203 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.199+0000 7f88f0bd9700 1 -- 172.21.15.135:0/805913362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 msgr2=0x7f88ec0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:22.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.199+0000 7f88f0bd9700 1 --2- 172.21.15.135:0/805913362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec0f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7f88dc009fb0 tx=0x7f88dc009b30).stop 2023-11-15T06:43:22.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.199+0000 7f88f0bd9700 1 -- 172.21.15.135:0/805913362 shutdown_connections 2023-11-15T06:43:22.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.199+0000 7f88f0bd9700 1 --2- 172.21.15.135:0/805913362 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec0f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:22.204 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.199+0000 7f88f0bd9700 1 -- 172.21.15.135:0/805913362 >> 172.21.15.135:0/805913362 conn(0x7f88ec0f0b70 msgr2=0x7f88ec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:22.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.199+0000 7f88f0bd9700 1 -- 172.21.15.135:0/805913362 shutdown_connections 2023-11-15T06:43:22.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.199+0000 7f88f0bd9700 1 -- 172.21.15.135:0/805913362 wait complete. 2023-11-15T06:43:22.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88f0bd9700 1 Processor -- start 2023-11-15T06:43:22.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88f0bd9700 1 -- start start 2023-11-15T06:43:22.205 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88f0bd9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:22.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88f0bd9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f88ec060220 con 0x7f88ec0f5410 2023-11-15T06:43:22.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88ea59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:22.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88ea59c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57782/0 (socket says 172.21.15.135:57782) 2023-11-15T06:43:22.206 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88ea59c700 1 -- 172.21.15.135:0/2622534132 learned_addr learned my addr 172.21.15.135:0/2622534132 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:22.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.200+0000 7f88ea59c700 1 -- 172.21.15.135:0/2622534132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f88dc005040 con 0x7f88ec0f5410 2023-11-15T06:43:22.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88ea59c700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec109450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f88dc013510 tx=0x7f88dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:22.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88e2ffd700 1 -- 172.21.15.135:0/2622534132 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88dc009320 con 0x7f88ec0f5410 2023-11-15T06:43:22.207 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88e2ffd700 1 -- 172.21.15.135:0/2622534132 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f88dc013f00 con 0x7f88ec0f5410 2023-11-15T06:43:22.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f88ec060420 con 0x7f88ec0f5410 2023-11-15T06:43:22.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88e2ffd700 1 -- 172.21.15.135:0/2622534132 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f88dc014d90 con 0x7f88ec0f5410 2023-11-15T06:43:22.208 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f88ec109b00 con 0x7f88ec0f5410 2023-11-15T06:43:22.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88e2ffd700 1 -- 172.21.15.135:0/2622534132 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f88dc0095f0 con 0x7f88ec0f5410 2023-11-15T06:43:22.209 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.201+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f88ec059100 con 0x7f88ec0f5410 2023-11-15T06:43:22.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.202+0000 7f88e2ffd700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f88d802ff60 0x7f88d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:22.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.202+0000 7f88e2ffd700 1 -- 172.21.15.135:0/2622534132 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f88dc041000 con 0x7f88ec0f5410 2023-11-15T06:43:22.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.202+0000 7f88e9d9b700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f88d802ff60 0x7f88d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:22.210 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.202+0000 7f88e9d9b700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f88d802ff60 0x7f88d8032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f88d40096a0 tx=0x7f88d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:22.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.207+0000 7f88e2ffd700 1 -- 172.21.15.135:0/2622534132 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f88dc014420 con 0x7f88ec0f5410 2023-11-15T06:43:22.413 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.411+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f88ec045ad0 con 0x7f88ec0f5410 2023-11-15T06:43:22.414 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.412+0000 7f88e2ffd700 1 -- 172.21.15.135:0/2622534132 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f88dc01d7f0 con 0x7f88ec0f5410 2023-11-15T06:43:22.415 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:22.415 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:22.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f88d802ff60 msgr2=0x7f88d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:22.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f88d802ff60 0x7f88d8032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7f88d40096a0 tx=0x7f88d4006b40).stop 2023-11-15T06:43:22.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 msgr2=0x7f88ec109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:22.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec109450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7f88dc013510 tx=0x7f88dc00b040).stop 2023-11-15T06:43:22.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 shutdown_connections 2023-11-15T06:43:22.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f88d802ff60 0x7f88d8032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:22.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 --2- 172.21.15.135:0/2622534132 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f88ec0f5410 0x7f88ec109450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:22.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 >> 172.21.15.135:0/2622534132 conn(0x7f88ec0f0b70 msgr2=0x7f88ec0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:22.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 shutdown_connections 2023-11-15T06:43:22.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:22.413+0000 7f88f0bd9700 1 -- 172.21.15.135:0/2622534132 wait complete. 2023-11-15T06:43:22.418 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:22.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:22 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:22.412674+0000 mon.smithi123 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.135:0/2622534132' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:23.803 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:23.803 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:23.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:22.283869+0000 mgr.smithi123.razjtn (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:25.001 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.997+0000 7f379855d700 1 -- 172.21.15.135:0/2905716493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 msgr2=0x7f37900f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:25.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.997+0000 7f379855d700 1 --2- 172.21.15.135:0/2905716493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f37900f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f3780009fb0 tx=0x7f3780009b30).stop 2023-11-15T06:43:25.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 -- 172.21.15.135:0/2905716493 shutdown_connections 2023-11-15T06:43:25.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 --2- 172.21.15.135:0/2905716493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f37900f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:25.002 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 -- 172.21.15.135:0/2905716493 >> 172.21.15.135:0/2905716493 conn(0x7f37900f0b70 msgr2=0x7f37900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:25.003 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 -- 172.21.15.135:0/2905716493 shutdown_connections 2023-11-15T06:43:25.003 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 -- 172.21.15.135:0/2905716493 wait complete. 2023-11-15T06:43:25.003 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 Processor -- start 2023-11-15T06:43:25.003 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 -- start start 2023-11-15T06:43:25.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f3790106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:25.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.998+0000 7f379855d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3790106c20 con 0x7f37900f5410 2023-11-15T06:43:25.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f37962f9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f3790106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:25.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f37962f9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f3790106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57802/0 (socket says 172.21.15.135:57802) 2023-11-15T06:43:25.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f37962f9700 1 -- 172.21.15.135:0/1361227711 learned_addr learned my addr 172.21.15.135:0/1361227711 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:25.004 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f37962f9700 1 -- 172.21.15.135:0/1361227711 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3780005040 con 0x7f37900f5410 2023-11-15T06:43:25.005 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f37962f9700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f3790106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f3780013510 tx=0x7f378000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:25.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f3786ffd700 1 -- 172.21.15.135:0/1361227711 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3780009320 con 0x7f37900f5410 2023-11-15T06:43:25.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3790106e20 con 0x7f37900f5410 2023-11-15T06:43:25.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f3786ffd700 1 -- 172.21.15.135:0/1361227711 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3780013f00 con 0x7f37900f5410 2023-11-15T06:43:25.006 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f3786ffd700 1 -- 172.21.15.135:0/1361227711 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3780014d90 con 0x7f37900f5410 2023-11-15T06:43:25.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:24.999+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37901072b0 con 0x7f37900f5410 2023-11-15T06:43:25.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.000+0000 7f3786ffd700 1 -- 172.21.15.135:0/1361227711 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37800263a0 con 0x7f37900f5410 2023-11-15T06:43:25.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.000+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3790045ad0 con 0x7f37900f5410 2023-11-15T06:43:25.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.000+0000 7f3786ffd700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f377c02ff60 0x7f377c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:25.007 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.000+0000 7f3786ffd700 1 -- 172.21.15.135:0/1361227711 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37800413a0 con 0x7f37900f5410 2023-11-15T06:43:25.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.000+0000 7f3795af8700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f377c02ff60 0x7f377c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:25.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.001+0000 7f3795af8700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f377c02ff60 0x7f377c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f378c0096a0 tx=0x7f378c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:25.008 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.005+0000 7f3786ffd700 1 -- 172.21.15.135:0/1361227711 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f378001d3d0 con 0x7f37900f5410 2023-11-15T06:43:25.211 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.209+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37900fae60 con 0x7f37900f5410 2023-11-15T06:43:25.212 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.209+0000 7f3786ffd700 1 -- 172.21.15.135:0/1361227711 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3780012330 con 0x7f37900f5410 2023-11-15T06:43:25.212 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:25.213 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:25.214 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f377c02ff60 msgr2=0x7f377c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:25.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f377c02ff60 0x7f377c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f378c0096a0 tx=0x7f378c006b40).stop 2023-11-15T06:43:25.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 msgr2=0x7f3790106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:25.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f3790106730 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f3780013510 tx=0x7f378000b040).stop 2023-11-15T06:43:25.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 shutdown_connections 2023-11-15T06:43:25.215 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f377c02ff60 0x7f377c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:25.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 --2- 172.21.15.135:0/1361227711 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f37900f5410 0x7f3790106730 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:25.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 >> 172.21.15.135:0/1361227711 conn(0x7f37900f0b70 msgr2=0x7f37900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:25.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 shutdown_connections 2023-11-15T06:43:25.216 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:25.211+0000 7f379855d700 1 -- 172.21.15.135:0/1361227711 wait complete. 2023-11-15T06:43:25.216 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:24.284112+0000 mgr.smithi123.razjtn (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:25.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:25.210537+0000 mon.smithi123 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.135:0/1361227711' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:26.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:26.583 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:27.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.725+0000 7f8987aa6700 1 -- 172.21.15.135:0/776329838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 msgr2=0x7f89800f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:27.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.725+0000 7f8987aa6700 1 --2- 172.21.15.135:0/776329838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89800f90f0 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7f8970009fb0 tx=0x7f8970009b30).stop 2023-11-15T06:43:27.730 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.725+0000 7f8987aa6700 1 -- 172.21.15.135:0/776329838 shutdown_connections 2023-11-15T06:43:27.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.725+0000 7f8987aa6700 1 --2- 172.21.15.135:0/776329838 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89800f90f0 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:27.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.725+0000 7f8987aa6700 1 -- 172.21.15.135:0/776329838 >> 172.21.15.135:0/776329838 conn(0x7f89800f0b70 msgr2=0x7f89800f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:27.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.725+0000 7f8987aa6700 1 -- 172.21.15.135:0/776329838 shutdown_connections 2023-11-15T06:43:27.731 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.725+0000 7f8987aa6700 1 -- 172.21.15.135:0/776329838 wait complete. 2023-11-15T06:43:27.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8987aa6700 1 Processor -- start 2023-11-15T06:43:27.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8987aa6700 1 -- start start 2023-11-15T06:43:27.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8987aa6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89801066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:27.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8987aa6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8980106bb0 con 0x7f8980060280 2023-11-15T06:43:27.732 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8985842700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89801066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:27.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8985842700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89801066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:57828/0 (socket says 172.21.15.135:57828) 2023-11-15T06:43:27.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8985842700 1 -- 172.21.15.135:0/3374040530 learned_addr learned my addr 172.21.15.135:0/3374040530 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:27.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.726+0000 7f8985842700 1 -- 172.21.15.135:0/3374040530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8970005040 con 0x7f8980060280 2023-11-15T06:43:27.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.727+0000 7f8985842700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89801066c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f8970013510 tx=0x7f897000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:27.733 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.727+0000 7f89767fc700 1 -- 172.21.15.135:0/3374040530 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8970009320 con 0x7f8980060280 2023-11-15T06:43:27.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.727+0000 7f89767fc700 1 -- 172.21.15.135:0/3374040530 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8970013f00 con 0x7f8980060280 2023-11-15T06:43:27.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.727+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8980106db0 con 0x7f8980060280 2023-11-15T06:43:27.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.727+0000 7f89767fc700 1 -- 172.21.15.135:0/3374040530 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8970014dc0 con 0x7f8980060280 2023-11-15T06:43:27.734 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.727+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8980107240 con 0x7f8980060280 2023-11-15T06:43:27.736 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.728+0000 7f89767fc700 1 -- 172.21.15.135:0/3374040530 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8970014420 con 0x7f8980060280 2023-11-15T06:43:27.736 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.728+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8980045ad0 con 0x7f8980060280 2023-11-15T06:43:27.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.728+0000 7f89767fc700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f896c02ff60 0x7f896c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:27.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.728+0000 7f89767fc700 1 -- 172.21.15.135:0/3374040530 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89700410b0 con 0x7f8980060280 2023-11-15T06:43:27.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.728+0000 7f8985041700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f896c02ff60 0x7f896c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:27.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.728+0000 7f8985041700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f896c02ff60 0x7f896c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f897c0096a0 tx=0x7f897c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:27.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.732+0000 7f89767fc700 1 -- 172.21.15.135:0/3374040530 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89700095f0 con 0x7f8980060280 2023-11-15T06:43:27.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:26.284363+0000 mgr.smithi123.razjtn (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:27.942 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.939+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f89801080f0 con 0x7f8980060280 2023-11-15T06:43:27.943 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.940+0000 7f89767fc700 1 -- 172.21.15.135:0/3374040530 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f897001cb10 con 0x7f8980060280 2023-11-15T06:43:27.943 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:27.943 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:27.944 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f896c02ff60 msgr2=0x7f896c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:27.945 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f896c02ff60 0x7f896c032420 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7f897c0096a0 tx=0x7f897c006b40).stop 2023-11-15T06:43:27.945 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 msgr2=0x7f89801066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:27.945 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89801066c0 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7f8970013510 tx=0x7f897000b040).stop 2023-11-15T06:43:27.945 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 shutdown_connections 2023-11-15T06:43:27.945 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f896c02ff60 0x7f896c032420 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:27.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 --2- 172.21.15.135:0/3374040530 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8980060280 0x7f89801066c0 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:27.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 >> 172.21.15.135:0/3374040530 conn(0x7f89800f0b70 msgr2=0x7f89800f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:27.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 shutdown_connections 2023-11-15T06:43:27.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:27.941+0000 7f8987aa6700 1 -- 172.21.15.135:0/3374040530 wait complete. 2023-11-15T06:43:27.946 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:28.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:28 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:27.940818+0000 mon.smithi123 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.135:0/3374040530' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:29.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:29.328 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:29.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:28.284632+0000 mgr.smithi123.razjtn (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:30.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.457+0000 7f7d700aa700 1 -- 172.21.15.135:0/3070549203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 msgr2=0x7f7d680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:30.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.457+0000 7f7d700aa700 1 --2- 172.21.15.135:0/3070549203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d680f5830 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f7d58009fb0 tx=0x7f7d58009b30).stop 2023-11-15T06:43:30.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.457+0000 7f7d700aa700 1 -- 172.21.15.135:0/3070549203 shutdown_connections 2023-11-15T06:43:30.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.457+0000 7f7d700aa700 1 --2- 172.21.15.135:0/3070549203 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d680f5830 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:30.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.457+0000 7f7d700aa700 1 -- 172.21.15.135:0/3070549203 >> 172.21.15.135:0/3070549203 conn(0x7f7d680f0b70 msgr2=0x7f7d680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:30.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.457+0000 7f7d700aa700 1 -- 172.21.15.135:0/3070549203 shutdown_connections 2023-11-15T06:43:30.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.457+0000 7f7d700aa700 1 -- 172.21.15.135:0/3070549203 wait complete. 2023-11-15T06:43:30.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.458+0000 7f7d700aa700 1 Processor -- start 2023-11-15T06:43:30.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.458+0000 7f7d700aa700 1 -- start start 2023-11-15T06:43:30.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.458+0000 7f7d700aa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d68105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:30.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.458+0000 7f7d700aa700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d68105b20 con 0x7f7d680f5410 2023-11-15T06:43:30.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.458+0000 7f7d6de46700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d68105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:30.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.458+0000 7f7d6de46700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d68105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48800/0 (socket says 172.21.15.135:48800) 2023-11-15T06:43:30.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.458+0000 7f7d6de46700 1 -- 172.21.15.135:0/1249194507 learned_addr learned my addr 172.21.15.135:0/1249194507 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:30.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.459+0000 7f7d6de46700 1 -- 172.21.15.135:0/1249194507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d58005040 con 0x7f7d680f5410 2023-11-15T06:43:30.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.459+0000 7f7d6de46700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d68105630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7d58013510 tx=0x7f7d5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:30.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.459+0000 7f7d5e7fc700 1 -- 172.21.15.135:0/1249194507 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d58009320 con 0x7f7d680f5410 2023-11-15T06:43:30.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.459+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d68105d20 con 0x7f7d680f5410 2023-11-15T06:43:30.466 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.459+0000 7f7d5e7fc700 1 -- 172.21.15.135:0/1249194507 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d58013f00 con 0x7f7d680f5410 2023-11-15T06:43:30.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.459+0000 7f7d5e7fc700 1 -- 172.21.15.135:0/1249194507 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d58014d90 con 0x7f7d680f5410 2023-11-15T06:43:30.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.459+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d681061b0 con 0x7f7d680f5410 2023-11-15T06:43:30.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.460+0000 7f7d5e7fc700 1 -- 172.21.15.135:0/1249194507 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d580095f0 con 0x7f7d680f5410 2023-11-15T06:43:30.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.460+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d68045ad0 con 0x7f7d680f5410 2023-11-15T06:43:30.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.460+0000 7f7d5e7fc700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d5402ff60 0x7f7d54032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:30.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.460+0000 7f7d5e7fc700 1 -- 172.21.15.135:0/1249194507 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d58042db0 con 0x7f7d680f5410 2023-11-15T06:43:30.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.460+0000 7f7d6d645700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d5402ff60 0x7f7d54032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:30.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.461+0000 7f7d6d645700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d5402ff60 0x7f7d54032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f7d640096a0 tx=0x7f7d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:30.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.465+0000 7f7d5e7fc700 1 -- 172.21.15.135:0/1249194507 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d58014420 con 0x7f7d680f5410 2023-11-15T06:43:30.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.672+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d680fae60 con 0x7f7d680f5410 2023-11-15T06:43:30.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.673+0000 7f7d5e7fc700 1 -- 172.21.15.135:0/1249194507 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d58011290 con 0x7f7d680f5410 2023-11-15T06:43:30.676 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:30.676 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:30.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d5402ff60 msgr2=0x7f7d54032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:30.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d5402ff60 0x7f7d54032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f7d640096a0 tx=0x7f7d64006b40).stop 2023-11-15T06:43:30.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 msgr2=0x7f7d68105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:30.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d68105630 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f7d58013510 tx=0x7f7d5800b040).stop 2023-11-15T06:43:30.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 shutdown_connections 2023-11-15T06:43:30.678 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d5402ff60 0x7f7d54032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:30.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 --2- 172.21.15.135:0/1249194507 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d680f5410 0x7f7d68105630 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:30.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 >> 172.21.15.135:0/1249194507 conn(0x7f7d680f0b70 msgr2=0x7f7d680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:30.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 shutdown_connections 2023-11-15T06:43:30.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:30.675+0000 7f7d700aa700 1 -- 172.21.15.135:0/1249194507 wait complete. 2023-11-15T06:43:30.680 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:31.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:30.284912+0000 mgr.smithi123.razjtn (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:31.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:30.673981+0000 mon.smithi123 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.135:0/1249194507' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:32.104 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:32.105 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:33.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 -- 172.21.15.135:0/1538052911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 msgr2=0x7fb2f00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:33.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 --2- 172.21.15.135:0/1538052911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f00f6930 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fb2e4009fb0 tx=0x7fb2e4009b30).stop 2023-11-15T06:43:33.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 -- 172.21.15.135:0/1538052911 shutdown_connections 2023-11-15T06:43:33.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 --2- 172.21.15.135:0/1538052911 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f00f6930 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:33.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 -- 172.21.15.135:0/1538052911 >> 172.21.15.135:0/1538052911 conn(0x7fb2f00f1c70 msgr2=0x7fb2f00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:33.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 -- 172.21.15.135:0/1538052911 shutdown_connections 2023-11-15T06:43:33.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 -- 172.21.15.135:0/1538052911 wait complete. 2023-11-15T06:43:33.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 Processor -- start 2023-11-15T06:43:33.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 -- start start 2023-11-15T06:43:33.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:33.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.231+0000 7fb2f7a46700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb2f0106c20 con 0x7fb2f00f6510 2023-11-15T06:43:33.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2f57e2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:33.237 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2f57e2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48830/0 (socket says 172.21.15.135:48830) 2023-11-15T06:43:33.237 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2f57e2700 1 -- 172.21.15.135:0/2367273880 learned_addr learned my addr 172.21.15.135:0/2367273880 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:33.238 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2f57e2700 1 -- 172.21.15.135:0/2367273880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb2e4005040 con 0x7fb2f00f6510 2023-11-15T06:43:33.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2f57e2700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f0106730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb2e4013510 tx=0x7fb2e400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:33.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2e1ffb700 1 -- 172.21.15.135:0/2367273880 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e4009320 con 0x7fb2f00f6510 2023-11-15T06:43:33.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb2f0106e20 con 0x7fb2f00f6510 2023-11-15T06:43:33.239 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2e1ffb700 1 -- 172.21.15.135:0/2367273880 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb2e4013f00 con 0x7fb2f00f6510 2023-11-15T06:43:33.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2e1ffb700 1 -- 172.21.15.135:0/2367273880 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb2e4014d90 con 0x7fb2f00f6510 2023-11-15T06:43:33.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.232+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb2f01072b0 con 0x7fb2f00f6510 2023-11-15T06:43:33.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.233+0000 7fb2e1ffb700 1 -- 172.21.15.135:0/2367273880 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb2e40095f0 con 0x7fb2f00f6510 2023-11-15T06:43:33.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.233+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb2f0045ad0 con 0x7fb2f00f6510 2023-11-15T06:43:33.240 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.233+0000 7fb2e1ffb700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb2dc02ff60 0x7fb2dc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:33.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.233+0000 7fb2e1ffb700 1 -- 172.21.15.135:0/2367273880 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb2e40410c0 con 0x7fb2f00f6510 2023-11-15T06:43:33.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.234+0000 7fb2f4fe1700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb2dc02ff60 0x7fb2dc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:33.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.234+0000 7fb2f4fe1700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb2dc02ff60 0x7fb2dc032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb2ec0096a0 tx=0x7fb2ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:33.241 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.238+0000 7fb2e1ffb700 1 -- 172.21.15.135:0/2367273880 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb2e401d7f0 con 0x7fb2f00f6510 2023-11-15T06:43:33.449 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.446+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb2f00fbf60 con 0x7fb2f00f6510 2023-11-15T06:43:33.450 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.447+0000 7fb2e1ffb700 1 -- 172.21.15.135:0/2367273880 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb2e4010230 con 0x7fb2f00f6510 2023-11-15T06:43:33.450 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:33.450 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:33.451 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb2dc02ff60 msgr2=0x7fb2dc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:33.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb2dc02ff60 0x7fb2dc032420 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fb2ec0096a0 tx=0x7fb2ec006b40).stop 2023-11-15T06:43:33.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 msgr2=0x7fb2f0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:33.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f0106730 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fb2e4013510 tx=0x7fb2e400b040).stop 2023-11-15T06:43:33.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 shutdown_connections 2023-11-15T06:43:33.452 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb2dc02ff60 0x7fb2dc032420 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:33.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 --2- 172.21.15.135:0/2367273880 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb2f00f6510 0x7fb2f0106730 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:33.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 >> 172.21.15.135:0/2367273880 conn(0x7fb2f00f1c70 msgr2=0x7fb2f00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:33.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 shutdown_connections 2023-11-15T06:43:33.453 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:33.449+0000 7fb2f7a46700 1 -- 172.21.15.135:0/2367273880 wait complete. 2023-11-15T06:43:33.454 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:33.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:32.285266+0000 mgr.smithi123.razjtn (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:33.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:33.448159+0000 mon.smithi123 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.135:0/2367273880' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:34.832 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:34.833 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:35.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:34.285665+0000 mgr.smithi123.razjtn (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:35.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/2753748502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 msgr2=0x7f1c040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:35.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 --2- 172.21.15.135:0/2753748502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c040f6930 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7f1bf4009fb0 tx=0x7f1bf4009b30).stop 2023-11-15T06:43:35.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/2753748502 shutdown_connections 2023-11-15T06:43:35.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 --2- 172.21.15.135:0/2753748502 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c040f6930 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:35.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/2753748502 >> 172.21.15.135:0/2753748502 conn(0x7f1c040f1c70 msgr2=0x7f1c040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:35.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/2753748502 shutdown_connections 2023-11-15T06:43:35.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/2753748502 wait complete. 2023-11-15T06:43:35.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.946+0000 7f1c0a6b2700 1 Processor -- start 2023-11-15T06:43:35.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.947+0000 7f1c0a6b2700 1 -- start start 2023-11-15T06:43:35.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.947+0000 7f1c0a6b2700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c04106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:35.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.947+0000 7f1c0a6b2700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1c04106c20 con 0x7f1c040f6510 2023-11-15T06:43:35.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.947+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c04106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:35.954 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.947+0000 7f1c03fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c04106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48858/0 (socket says 172.21.15.135:48858) 2023-11-15T06:43:35.954 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.947+0000 7f1c03fff700 1 -- 172.21.15.135:0/1259691277 learned_addr learned my addr 172.21.15.135:0/1259691277 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:35.954 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.947+0000 7f1c03fff700 1 -- 172.21.15.135:0/1259691277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1bf4005040 con 0x7f1c040f6510 2023-11-15T06:43:35.954 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.948+0000 7f1c03fff700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c04106730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f1bf4014000 tx=0x7f1bf400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:35.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.948+0000 7f1c00ff9700 1 -- 172.21.15.135:0/1259691277 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bf40095f0 con 0x7f1c040f6510 2023-11-15T06:43:35.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.948+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1c04106e20 con 0x7f1c040f6510 2023-11-15T06:43:35.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.948+0000 7f1c00ff9700 1 -- 172.21.15.135:0/1259691277 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1bf4007e70 con 0x7f1c040f6510 2023-11-15T06:43:35.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.948+0000 7f1c00ff9700 1 -- 172.21.15.135:0/1259691277 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1bf4014ea0 con 0x7f1c040f6510 2023-11-15T06:43:35.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.948+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1c041072b0 con 0x7f1c040f6510 2023-11-15T06:43:35.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.949+0000 7f1c00ff9700 1 -- 172.21.15.135:0/1259691277 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1bf4025070 con 0x7f1c040f6510 2023-11-15T06:43:35.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.949+0000 7f1c00ff9700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1bec02ff60 0x7f1bec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:35.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.949+0000 7f1c00ff9700 1 -- 172.21.15.135:0/1259691277 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1bf40409f0 con 0x7f1c040f6510 2023-11-15T06:43:35.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.949+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1c04045ad0 con 0x7f1c040f6510 2023-11-15T06:43:35.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.949+0000 7f1c037fe700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1bec02ff60 0x7f1bec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:35.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.949+0000 7f1c037fe700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1bec02ff60 0x7f1bec032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1bf80096a0 tx=0x7f1bf8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:35.957 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:35.953+0000 7f1c00ff9700 1 -- 172.21.15.135:0/1259691277 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1bf4026350 con 0x7f1c040f6510 2023-11-15T06:43:36.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.161+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1c040fc050 con 0x7f1c040f6510 2023-11-15T06:43:36.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.162+0000 7f1c00ff9700 1 -- 172.21.15.135:0/1259691277 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1bf4025300 con 0x7f1c040f6510 2023-11-15T06:43:36.165 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:36.165 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:36.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1bec02ff60 msgr2=0x7f1bec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:36.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1bec02ff60 0x7f1bec032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7f1bf80096a0 tx=0x7f1bf8006b40).stop 2023-11-15T06:43:36.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 msgr2=0x7f1c04106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:36.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c04106730 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7f1bf4014000 tx=0x7f1bf400b040).stop 2023-11-15T06:43:36.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 shutdown_connections 2023-11-15T06:43:36.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1bec02ff60 0x7f1bec032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:36.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 --2- 172.21.15.135:0/1259691277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1c040f6510 0x7f1c04106730 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 >> 172.21.15.135:0/1259691277 conn(0x7f1c040f1c70 msgr2=0x7f1c040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 shutdown_connections 2023-11-15T06:43:36.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:36.163+0000 7f1c0a6b2700 1 -- 172.21.15.135:0/1259691277 wait complete. 2023-11-15T06:43:36.170 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:36.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:36 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:36.162835+0000 mon.smithi123 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.135:0/1259691277' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:37.565 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:37.565 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:37.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:36.285907+0000 mgr.smithi123.razjtn (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:38.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.674+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3302741623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 msgr2=0x7fe4540f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:38.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.674+0000 7fe45b9e8700 1 --2- 172.21.15.135:0/3302741623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe4540f5830 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7fe450009fb0 tx=0x7fe450009b30).stop 2023-11-15T06:43:38.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.674+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3302741623 shutdown_connections 2023-11-15T06:43:38.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.674+0000 7fe45b9e8700 1 --2- 172.21.15.135:0/3302741623 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe4540f5830 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:38.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.674+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3302741623 >> 172.21.15.135:0/3302741623 conn(0x7fe4540f0b70 msgr2=0x7fe4540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:38.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3302741623 shutdown_connections 2023-11-15T06:43:38.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3302741623 wait complete. 2023-11-15T06:43:38.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe45b9e8700 1 Processor -- start 2023-11-15T06:43:38.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe45b9e8700 1 -- start start 2023-11-15T06:43:38.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe45b9e8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe45410b820 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:38.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe45b9e8700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe454107210 con 0x7fe4540f5410 2023-11-15T06:43:38.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe459784700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe45410b820 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:38.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe459784700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe45410b820 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36628/0 (socket says 172.21.15.135:36628) 2023-11-15T06:43:38.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.675+0000 7fe459784700 1 -- 172.21.15.135:0/3999051397 learned_addr learned my addr 172.21.15.135:0/3999051397 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:38.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.676+0000 7fe459784700 1 -- 172.21.15.135:0/3999051397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe450005040 con 0x7fe4540f5410 2023-11-15T06:43:38.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.676+0000 7fe459784700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe45410b820 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe450013510 tx=0x7fe45000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:38.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.676+0000 7fe445ffb700 1 -- 172.21.15.135:0/3999051397 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe450009320 con 0x7fe4540f5410 2023-11-15T06:43:38.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.676+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe454060220 con 0x7fe4540f5410 2023-11-15T06:43:38.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.676+0000 7fe445ffb700 1 -- 172.21.15.135:0/3999051397 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe450013f00 con 0x7fe4540f5410 2023-11-15T06:43:38.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.676+0000 7fe445ffb700 1 -- 172.21.15.135:0/3999051397 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe450014d90 con 0x7fe4540f5410 2023-11-15T06:43:38.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.676+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe45410bd10 con 0x7fe4540f5410 2023-11-15T06:43:38.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.677+0000 7fe445ffb700 1 -- 172.21.15.135:0/3999051397 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe4500263a0 con 0x7fe4540f5410 2023-11-15T06:43:38.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.677+0000 7fe445ffb700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe44002ffb0 0x7fe440032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:38.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.677+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe454045ad0 con 0x7fe4540f5410 2023-11-15T06:43:38.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.677+0000 7fe445ffb700 1 -- 172.21.15.135:0/3999051397 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe4500410c0 con 0x7fe4540f5410 2023-11-15T06:43:38.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.677+0000 7fe458f83700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe44002ffb0 0x7fe440032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:38.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.678+0000 7fe458f83700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe44002ffb0 0x7fe440032470 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe4480096a0 tx=0x7fe448006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:38.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.682+0000 7fe445ffb700 1 -- 172.21.15.135:0/3999051397 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe450012920 con 0x7fe4540f5410 2023-11-15T06:43:38.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.888+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe4540faf50 con 0x7fe4540f5410 2023-11-15T06:43:38.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.888+0000 7fe445ffb700 1 -- 172.21.15.135:0/3999051397 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe45002b020 con 0x7fe4540f5410 2023-11-15T06:43:38.892 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:38.892 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:38.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe44002ffb0 msgr2=0x7fe440032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:38.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe44002ffb0 0x7fe440032470 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7fe4480096a0 tx=0x7fe448006b40).stop 2023-11-15T06:43:38.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 msgr2=0x7fe45410b820 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:38.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe45410b820 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7fe450013510 tx=0x7fe45000b040).stop 2023-11-15T06:43:38.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 shutdown_connections 2023-11-15T06:43:38.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe44002ffb0 0x7fe440032470 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:38.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 --2- 172.21.15.135:0/3999051397 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe4540f5410 0x7fe45410b820 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:38.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 >> 172.21.15.135:0/3999051397 conn(0x7fe4540f0b70 msgr2=0x7fe4540f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:38.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 shutdown_connections 2023-11-15T06:43:38.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:38.890+0000 7fe45b9e8700 1 -- 172.21.15.135:0/3999051397 wait complete. 2023-11-15T06:43:38.895 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:39.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:38.286193+0000 mgr.smithi123.razjtn (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:39.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:38.889426+0000 mon.smithi123 (mon.0) 255 : audit [DBG] from='client.? 172.21.15.135:0/3999051397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:40.278 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:40.278 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:40.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:40.368671+0000 mon.smithi123 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:43:40.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:40.374369+0000 mon.smithi123 (mon.0) 257 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:43:41.374 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.370+0000 7fbf79738700 1 -- 172.21.15.135:0/1395172206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 msgr2=0x7fbf740f6df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:41.375 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.370+0000 7fbf79738700 1 --2- 172.21.15.135:0/1395172206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf740f6df0 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7fbf68009fb0 tx=0x7fbf68009b30).stop 2023-11-15T06:43:41.375 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.370+0000 7fbf79738700 1 -- 172.21.15.135:0/1395172206 shutdown_connections 2023-11-15T06:43:41.375 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.370+0000 7fbf79738700 1 --2- 172.21.15.135:0/1395172206 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf740f6df0 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:41.375 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.370+0000 7fbf79738700 1 -- 172.21.15.135:0/1395172206 >> 172.21.15.135:0/1395172206 conn(0x7fbf740f02e0 msgr2=0x7fbf740f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:41.375 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.370+0000 7fbf79738700 1 -- 172.21.15.135:0/1395172206 shutdown_connections 2023-11-15T06:43:41.376 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.370+0000 7fbf79738700 1 -- 172.21.15.135:0/1395172206 wait complete. 2023-11-15T06:43:41.376 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.371+0000 7fbf79738700 1 Processor -- start 2023-11-15T06:43:41.376 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.371+0000 7fbf79738700 1 -- start start 2023-11-15T06:43:41.376 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.371+0000 7fbf79738700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf7410b7c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:41.376 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.371+0000 7fbf79738700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbf741071b0 con 0x7fbf740f69d0 2023-11-15T06:43:41.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.371+0000 7fbf72ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf7410b7c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:41.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.371+0000 7fbf72ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf7410b7c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36652/0 (socket says 172.21.15.135:36652) 2023-11-15T06:43:41.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf72ffd700 1 -- 172.21.15.135:0/4172449099 learned_addr learned my addr 172.21.15.135:0/4172449099 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:41.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf72ffd700 1 -- 172.21.15.135:0/4172449099 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbf68005040 con 0x7fbf740f69d0 2023-11-15T06:43:41.377 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf72ffd700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf7410b7c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fbf68014000 tx=0x7fbf6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:41.378 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf5b7fe700 1 -- 172.21.15.135:0/4172449099 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf680095f0 con 0x7fbf740f69d0 2023-11-15T06:43:41.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbf740f5250 con 0x7fbf740f69d0 2023-11-15T06:43:41.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf5b7fe700 1 -- 172.21.15.135:0/4172449099 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbf68007e70 con 0x7fbf740f69d0 2023-11-15T06:43:41.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf5b7fe700 1 -- 172.21.15.135:0/4172449099 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbf68014ea0 con 0x7fbf740f69d0 2023-11-15T06:43:41.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.372+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbf7410bcb0 con 0x7fbf740f69d0 2023-11-15T06:43:41.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.373+0000 7fbf5b7fe700 1 -- 172.21.15.135:0/4172449099 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbf6801d4e0 con 0x7fbf740f69d0 2023-11-15T06:43:41.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.373+0000 7fbf5b7fe700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbf5c02ffb0 0x7fbf5c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:41.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.373+0000 7fbf5b7fe700 1 -- 172.21.15.135:0/4172449099 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbf68040960 con 0x7fbf740f69d0 2023-11-15T06:43:41.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.373+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbf74045ad0 con 0x7fbf740f69d0 2023-11-15T06:43:41.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.373+0000 7fbf727fc700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbf5c02ffb0 0x7fbf5c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:41.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.374+0000 7fbf727fc700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbf5c02ffb0 0x7fbf5c032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fbf640096a0 tx=0x7fbf64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:41.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.378+0000 7fbf5b7fe700 1 -- 172.21.15.135:0/4172449099 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbf6800f8f0 con 0x7fbf740f69d0 2023-11-15T06:43:41.589 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.586+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbf7410c470 con 0x7fbf740f69d0 2023-11-15T06:43:41.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.587+0000 7fbf5b7fe700 1 -- 172.21.15.135:0/4172449099 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbf6802c090 con 0x7fbf740f69d0 2023-11-15T06:43:41.591 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:41.591 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:41.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbf5c02ffb0 msgr2=0x7fbf5c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:41.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbf5c02ffb0 0x7fbf5c032470 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7fbf640096a0 tx=0x7fbf64006b40).stop 2023-11-15T06:43:41.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 msgr2=0x7fbf7410b7c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:41.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf7410b7c0 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7fbf68014000 tx=0x7fbf6800b040).stop 2023-11-15T06:43:41.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 shutdown_connections 2023-11-15T06:43:41.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbf5c02ffb0 0x7fbf5c032470 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:41.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 --2- 172.21.15.135:0/4172449099 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbf740f69d0 0x7fbf7410b7c0 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:41.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 >> 172.21.15.135:0/4172449099 conn(0x7fbf740f02e0 msgr2=0x7fbf740f0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:41.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 shutdown_connections 2023-11-15T06:43:41.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:41.589+0000 7fbf79738700 1 -- 172.21.15.135:0/4172449099 wait complete. 2023-11-15T06:43:41.594 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:41.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:40.286481+0000 mgr.smithi123.razjtn (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:42.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:42 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:41.588406+0000 mon.smithi123 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.135:0/4172449099' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:42.966 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:42.967 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:43.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:42.286828+0000 mgr.smithi123.razjtn (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:44.076 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.069+0000 7f4c3c570700 1 -- 172.21.15.135:0/1901064346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 msgr2=0x7f4c340f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:44.076 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.069+0000 7f4c3c570700 1 --2- 172.21.15.135:0/1901064346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c340f5830 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7f4c24009fb0 tx=0x7f4c24009b30).stop 2023-11-15T06:43:44.076 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.069+0000 7f4c3c570700 1 -- 172.21.15.135:0/1901064346 shutdown_connections 2023-11-15T06:43:44.077 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.069+0000 7f4c3c570700 1 --2- 172.21.15.135:0/1901064346 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c340f5830 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:44.077 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.069+0000 7f4c3c570700 1 -- 172.21.15.135:0/1901064346 >> 172.21.15.135:0/1901064346 conn(0x7f4c340f0b70 msgr2=0x7f4c340f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:44.077 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.069+0000 7f4c3c570700 1 -- 172.21.15.135:0/1901064346 shutdown_connections 2023-11-15T06:43:44.078 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.069+0000 7f4c3c570700 1 -- 172.21.15.135:0/1901064346 wait complete. 2023-11-15T06:43:44.078 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.070+0000 7f4c3c570700 1 Processor -- start 2023-11-15T06:43:44.078 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.070+0000 7f4c3c570700 1 -- start start 2023-11-15T06:43:44.078 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.070+0000 7f4c3c570700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c34105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:44.079 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.070+0000 7f4c3c570700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4c34105b20 con 0x7f4c340f5410 2023-11-15T06:43:44.079 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.070+0000 7f4c3a30c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c34105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:44.080 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.070+0000 7f4c3a30c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c34105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36674/0 (socket says 172.21.15.135:36674) 2023-11-15T06:43:44.080 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.070+0000 7f4c3a30c700 1 -- 172.21.15.135:0/3359928843 learned_addr learned my addr 172.21.15.135:0/3359928843 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:44.080 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.071+0000 7f4c3a30c700 1 -- 172.21.15.135:0/3359928843 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c24005040 con 0x7f4c340f5410 2023-11-15T06:43:44.081 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.071+0000 7f4c3a30c700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c34105630 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f4c24013510 tx=0x7f4c2400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:44.081 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.071+0000 7f4c2affd700 1 -- 172.21.15.135:0/3359928843 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c24009320 con 0x7f4c340f5410 2023-11-15T06:43:44.081 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.071+0000 7f4c2affd700 1 -- 172.21.15.135:0/3359928843 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c24013f00 con 0x7f4c340f5410 2023-11-15T06:43:44.082 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.071+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4c34105d20 con 0x7f4c340f5410 2023-11-15T06:43:44.082 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.071+0000 7f4c2affd700 1 -- 172.21.15.135:0/3359928843 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c24014d90 con 0x7f4c340f5410 2023-11-15T06:43:44.082 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.071+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4c341061b0 con 0x7f4c340f5410 2023-11-15T06:43:44.082 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.072+0000 7f4c2affd700 1 -- 172.21.15.135:0/3359928843 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c240095f0 con 0x7f4c340f5410 2023-11-15T06:43:44.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.072+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4c34045ad0 con 0x7f4c340f5410 2023-11-15T06:43:44.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.072+0000 7f4c2affd700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c2002ff60 0x7f4c20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:44.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.072+0000 7f4c2affd700 1 -- 172.21.15.135:0/3359928843 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c240410c0 con 0x7f4c340f5410 2023-11-15T06:43:44.083 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.072+0000 7f4c39b0b700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c2002ff60 0x7f4c20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:44.084 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.072+0000 7f4c39b0b700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c2002ff60 0x7f4c20032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4c300096a0 tx=0x7f4c30006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:44.084 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.077+0000 7f4c2affd700 1 -- 172.21.15.135:0/3359928843 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c2401d7f0 con 0x7f4c340f5410 2023-11-15T06:43:44.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.282+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4c340fae60 con 0x7f4c340f5410 2023-11-15T06:43:44.287 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.283+0000 7f4c2affd700 1 -- 172.21.15.135:0/3359928843 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4c24010230 con 0x7f4c340f5410 2023-11-15T06:43:44.288 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:44.288 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:44.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c2002ff60 msgr2=0x7f4c20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:44.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c2002ff60 0x7f4c20032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4c300096a0 tx=0x7f4c30006b40).stop 2023-11-15T06:43:44.289 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 msgr2=0x7f4c34105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:44.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c34105630 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7f4c24013510 tx=0x7f4c2400b040).stop 2023-11-15T06:43:44.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 shutdown_connections 2023-11-15T06:43:44.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4c2002ff60 0x7f4c20032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:44.290 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 --2- 172.21.15.135:0/3359928843 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4c340f5410 0x7f4c34105630 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:44.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 >> 172.21.15.135:0/3359928843 conn(0x7f4c340f0b70 msgr2=0x7f4c340f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:44.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 shutdown_connections 2023-11-15T06:43:44.291 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:44.285+0000 7f4c3c570700 1 -- 172.21.15.135:0/3359928843 wait complete. 2023-11-15T06:43:44.291 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:44.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:44 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:44.284285+0000 mon.smithi123 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.135:0/3359928843' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:45.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:45.672 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:45.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:44.287090+0000 mgr.smithi123.razjtn (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:46.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.839+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1867645486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 msgr2=0x7f9ce40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:46.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.839+0000 7f9ce9c13700 1 --2- 172.21.15.135:0/1867645486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce40f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7f9cd4009fb0 tx=0x7f9cd4009b30).stop 2023-11-15T06:43:46.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.839+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1867645486 shutdown_connections 2023-11-15T06:43:46.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.839+0000 7f9ce9c13700 1 --2- 172.21.15.135:0/1867645486 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce40f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:46.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.839+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1867645486 >> 172.21.15.135:0/1867645486 conn(0x7f9ce40f0b70 msgr2=0x7f9ce40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:46.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.839+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1867645486 shutdown_connections 2023-11-15T06:43:46.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.839+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1867645486 wait complete. 2023-11-15T06:43:46.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.840+0000 7f9ce9c13700 1 Processor -- start 2023-11-15T06:43:46.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.840+0000 7f9ce9c13700 1 -- start start 2023-11-15T06:43:46.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.840+0000 7f9ce9c13700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:46.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.840+0000 7f9ce9c13700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9ce4060240 con 0x7f9ce40f5410 2023-11-15T06:43:46.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.840+0000 7f9ce37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:46.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.840+0000 7f9ce37fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:36698/0 (socket says 172.21.15.135:36698) 2023-11-15T06:43:46.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.840+0000 7f9ce37fe700 1 -- 172.21.15.135:0/1251525220 learned_addr learned my addr 172.21.15.135:0/1251525220 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:46.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.841+0000 7f9ce37fe700 1 -- 172.21.15.135:0/1251525220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cd4005040 con 0x7f9ce40f5410 2023-11-15T06:43:46.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.841+0000 7f9ce37fe700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce4109430 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9cd4013510 tx=0x7f9cd400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:46.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.841+0000 7f9ccbfff700 1 -- 172.21.15.135:0/1251525220 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd4009320 con 0x7f9ce40f5410 2023-11-15T06:43:46.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.841+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9ce4060440 con 0x7f9ce40f5410 2023-11-15T06:43:46.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.841+0000 7f9ccbfff700 1 -- 172.21.15.135:0/1251525220 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cd4013f00 con 0x7f9ce40f5410 2023-11-15T06:43:46.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.841+0000 7f9ccbfff700 1 -- 172.21.15.135:0/1251525220 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cd4014d90 con 0x7f9ce40f5410 2023-11-15T06:43:46.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.841+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9ce4109ae0 con 0x7f9ce40f5410 2023-11-15T06:43:46.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.842+0000 7f9ccbfff700 1 -- 172.21.15.135:0/1251525220 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cd4014420 con 0x7f9ce40f5410 2023-11-15T06:43:46.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.842+0000 7f9ccbfff700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9ccc02ff60 0x7f9ccc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:46.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.842+0000 7f9ccbfff700 1 -- 172.21.15.135:0/1251525220 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cd4040fb0 con 0x7f9ce40f5410 2023-11-15T06:43:46.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.842+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9ce4045ad0 con 0x7f9ce40f5410 2023-11-15T06:43:46.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.842+0000 7f9ce2ffd700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9ccc02ff60 0x7f9ccc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:46.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.843+0000 7f9ce2ffd700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9ccc02ff60 0x7f9ccc032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9cd80096a0 tx=0x7f9cd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:46.851 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:46.847+0000 7f9ccbfff700 1 -- 172.21.15.135:0/1251525220 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cd40095f0 con 0x7f9ce40f5410 2023-11-15T06:43:47.054 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.052+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9ce40faf50 con 0x7f9ce40f5410 2023-11-15T06:43:47.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.052+0000 7f9ccbfff700 1 -- 172.21.15.135:0/1251525220 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cd401db70 con 0x7f9ce40f5410 2023-11-15T06:43:47.055 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:47.056 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:47.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9ccc02ff60 msgr2=0x7f9ccc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:47.057 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9ccc02ff60 0x7f9ccc032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f9cd80096a0 tx=0x7f9cd8006b40).stop 2023-11-15T06:43:47.057 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 msgr2=0x7f9ce4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:47.057 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce4109430 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7f9cd4013510 tx=0x7f9cd400b040).stop 2023-11-15T06:43:47.057 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 shutdown_connections 2023-11-15T06:43:47.058 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f9ccc02ff60 0x7f9ccc032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:47.058 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 --2- 172.21.15.135:0/1251525220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f9ce40f5410 0x7f9ce4109430 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:47.058 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 >> 172.21.15.135:0/1251525220 conn(0x7f9ce40f0b70 msgr2=0x7f9ce40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:47.058 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 shutdown_connections 2023-11-15T06:43:47.058 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:47.054+0000 7f9ce9c13700 1 -- 172.21.15.135:0/1251525220 wait complete. 2023-11-15T06:43:47.059 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:47.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:46.287394+0000 mgr.smithi123.razjtn (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:47.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:47.053405+0000 mon.smithi123 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.135:0/1251525220' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:48.442 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:48.443 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:49.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.576+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/880655691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 msgr2=0x7fb6a40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:49.580 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.576+0000 7fb6ab5d1700 1 --2- 172.21.15.135:0/880655691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a40f6930 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7fb694009fb0 tx=0x7fb694009b30).stop 2023-11-15T06:43:49.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.576+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/880655691 shutdown_connections 2023-11-15T06:43:49.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.576+0000 7fb6ab5d1700 1 --2- 172.21.15.135:0/880655691 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a40f6930 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:49.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.576+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/880655691 >> 172.21.15.135:0/880655691 conn(0x7fb6a40f1c70 msgr2=0x7fb6a40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:49.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.577+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/880655691 shutdown_connections 2023-11-15T06:43:49.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.577+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/880655691 wait complete. 2023-11-15T06:43:49.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.577+0000 7fb6ab5d1700 1 Processor -- start 2023-11-15T06:43:49.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.577+0000 7fb6ab5d1700 1 -- start start 2023-11-15T06:43:49.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.577+0000 7fb6ab5d1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:49.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb6ab5d1700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6a4106c20 con 0x7fb6a40f6510 2023-11-15T06:43:49.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb6a936d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:49.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb6a936d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51782/0 (socket says 172.21.15.135:51782) 2023-11-15T06:43:49.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb6a936d700 1 -- 172.21.15.135:0/3451677733 learned_addr learned my addr 172.21.15.135:0/3451677733 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:49.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb6a936d700 1 -- 172.21.15.135:0/3451677733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb694005040 con 0x7fb6a40f6510 2023-11-15T06:43:49.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb6a936d700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a4106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fb694013510 tx=0x7fb69400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:49.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb699ffb700 1 -- 172.21.15.135:0/3451677733 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb694009320 con 0x7fb6a40f6510 2023-11-15T06:43:49.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6a4106e20 con 0x7fb6a40f6510 2023-11-15T06:43:49.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb699ffb700 1 -- 172.21.15.135:0/3451677733 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb694013f00 con 0x7fb6a40f6510 2023-11-15T06:43:49.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.578+0000 7fb699ffb700 1 -- 172.21.15.135:0/3451677733 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb694014d90 con 0x7fb6a40f6510 2023-11-15T06:43:49.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.579+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6a41072b0 con 0x7fb6a40f6510 2023-11-15T06:43:49.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.579+0000 7fb699ffb700 1 -- 172.21.15.135:0/3451677733 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6940095f0 con 0x7fb6a40f6510 2023-11-15T06:43:49.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.579+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6a4045ad0 con 0x7fb6a40f6510 2023-11-15T06:43:49.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.579+0000 7fb699ffb700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb69002ff60 0x7fb690032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:49.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.580+0000 7fb699ffb700 1 -- 172.21.15.135:0/3451677733 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb694041080 con 0x7fb6a40f6510 2023-11-15T06:43:49.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.580+0000 7fb6a8b6c700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb69002ff60 0x7fb690032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:49.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.580+0000 7fb6a8b6c700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb69002ff60 0x7fb690032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb6a00096a0 tx=0x7fb6a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:49.589 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.584+0000 7fb699ffb700 1 -- 172.21.15.135:0/3451677733 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb694014420 con 0x7fb6a40f6510 2023-11-15T06:43:49.796 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.794+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6a40fbf60 con 0x7fb6a40f6510 2023-11-15T06:43:49.797 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.794+0000 7fb699ffb700 1 -- 172.21.15.135:0/3451677733 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb694010230 con 0x7fb6a40f6510 2023-11-15T06:43:49.797 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:49.798 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb69002ff60 msgr2=0x7fb690032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb69002ff60 0x7fb690032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7fb6a00096a0 tx=0x7fb6a0006b40).stop 2023-11-15T06:43:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 msgr2=0x7fb6a4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:49.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a4106730 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7fb694013510 tx=0x7fb69400b040).stop 2023-11-15T06:43:49.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 shutdown_connections 2023-11-15T06:43:49.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb69002ff60 0x7fb690032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:49.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 --2- 172.21.15.135:0/3451677733 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb6a40f6510 0x7fb6a4106730 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:49.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 >> 172.21.15.135:0/3451677733 conn(0x7fb6a40f1c70 msgr2=0x7fb6a40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:49.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 shutdown_connections 2023-11-15T06:43:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:49.796+0000 7fb6ab5d1700 1 -- 172.21.15.135:0/3451677733 wait complete. 2023-11-15T06:43:49.801 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:48.287720+0000 mgr.smithi123.razjtn (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:50.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:49.795539+0000 mon.smithi123 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.135:0/3451677733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:51.153 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:51.183 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:51.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:50.288120+0000 mgr.smithi123.razjtn (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:52.315 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.310+0000 7f692ce83700 1 -- 172.21.15.135:0/2526549594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 msgr2=0x7f69280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:52.315 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.310+0000 7f692ce83700 1 --2- 172.21.15.135:0/2526549594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f69280f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f6918009fb0 tx=0x7f6918009b30).stop 2023-11-15T06:43:52.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 -- 172.21.15.135:0/2526549594 shutdown_connections 2023-11-15T06:43:52.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 --2- 172.21.15.135:0/2526549594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f69280f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:52.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 -- 172.21.15.135:0/2526549594 >> 172.21.15.135:0/2526549594 conn(0x7f69280f1c70 msgr2=0x7f69280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:52.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 -- 172.21.15.135:0/2526549594 shutdown_connections 2023-11-15T06:43:52.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 -- 172.21.15.135:0/2526549594 wait complete. 2023-11-15T06:43:52.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 Processor -- start 2023-11-15T06:43:52.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 -- start start 2023-11-15T06:43:52.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f6928106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:52.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.311+0000 7f692ce83700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6928106c20 con 0x7f69280f6510 2023-11-15T06:43:52.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f692659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f6928106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:52.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f692659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f6928106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51806/0 (socket says 172.21.15.135:51806) 2023-11-15T06:43:52.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f692659c700 1 -- 172.21.15.135:0/3231726124 learned_addr learned my addr 172.21.15.135:0/3231726124 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:52.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f692659c700 1 -- 172.21.15.135:0/3231726124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6918005040 con 0x7f69280f6510 2023-11-15T06:43:52.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f692659c700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f6928106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6918013510 tx=0x7f691800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:52.319 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f691effd700 1 -- 172.21.15.135:0/3231726124 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6918009320 con 0x7f69280f6510 2023-11-15T06:43:52.319 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f691effd700 1 -- 172.21.15.135:0/3231726124 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6918013f00 con 0x7f69280f6510 2023-11-15T06:43:52.319 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6928106e20 con 0x7f69280f6510 2023-11-15T06:43:52.319 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f691effd700 1 -- 172.21.15.135:0/3231726124 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6918014d90 con 0x7f69280f6510 2023-11-15T06:43:52.320 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.312+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f69281072b0 con 0x7f69280f6510 2023-11-15T06:43:52.322 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.313+0000 7f691effd700 1 -- 172.21.15.135:0/3231726124 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f69180095f0 con 0x7f69280f6510 2023-11-15T06:43:52.323 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.313+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6928045ad0 con 0x7f69280f6510 2023-11-15T06:43:52.323 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.313+0000 7f691effd700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f691402ff60 0x7f6914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:52.323 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.313+0000 7f691effd700 1 -- 172.21.15.135:0/3231726124 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f69180410c0 con 0x7f69280f6510 2023-11-15T06:43:52.324 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.313+0000 7f6925d9b700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f691402ff60 0x7f6914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:52.324 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.314+0000 7f6925d9b700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f691402ff60 0x7f6914032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f69100096a0 tx=0x7f6910006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:52.324 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.318+0000 7f691effd700 1 -- 172.21.15.135:0/3231726124 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6918014420 con 0x7f69280f6510 2023-11-15T06:43:52.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.523+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f69280fbf60 con 0x7f69280f6510 2023-11-15T06:43:52.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.524+0000 7f691effd700 1 -- 172.21.15.135:0/3231726124 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6918010230 con 0x7f69280f6510 2023-11-15T06:43:52.527 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:52.527 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:52.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.525+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f691402ff60 msgr2=0x7f6914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:52.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.525+0000 7f692ce83700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f691402ff60 0x7f6914032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f69100096a0 tx=0x7f6910006b40).stop 2023-11-15T06:43:52.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.525+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 msgr2=0x7f6928106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:52.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.525+0000 7f692ce83700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f6928106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f6918013510 tx=0x7f691800b040).stop 2023-11-15T06:43:52.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.526+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 shutdown_connections 2023-11-15T06:43:52.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.526+0000 7f692ce83700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f691402ff60 0x7f6914032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:52.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.526+0000 7f692ce83700 1 --2- 172.21.15.135:0/3231726124 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f69280f6510 0x7f6928106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:52.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.526+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 >> 172.21.15.135:0/3231726124 conn(0x7f69280f1c70 msgr2=0x7f69280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:52.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.526+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 shutdown_connections 2023-11-15T06:43:52.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:52.526+0000 7f692ce83700 1 -- 172.21.15.135:0/3231726124 wait complete. 2023-11-15T06:43:52.531 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:53.906 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:53.906 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:53.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:52.288523+0000 mgr.smithi123.razjtn (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:53.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:52.524972+0000 mon.smithi123 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.135:0/3231726124' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:55.038 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.034+0000 7f1a17c7f700 1 -- 172.21.15.135:0/621346219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 msgr2=0x7f1a100f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:55.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.034+0000 7f1a17c7f700 1 --2- 172.21.15.135:0/621346219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a100f5830 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7f1a0c009fb0 tx=0x7f1a0c009b30).stop 2023-11-15T06:43:55.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.034+0000 7f1a17c7f700 1 -- 172.21.15.135:0/621346219 shutdown_connections 2023-11-15T06:43:55.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.034+0000 7f1a17c7f700 1 --2- 172.21.15.135:0/621346219 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a100f5830 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:55.039 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.034+0000 7f1a17c7f700 1 -- 172.21.15.135:0/621346219 >> 172.21.15.135:0/621346219 conn(0x7f1a100f0b70 msgr2=0x7f1a100f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:55.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.034+0000 7f1a17c7f700 1 -- 172.21.15.135:0/621346219 shutdown_connections 2023-11-15T06:43:55.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.034+0000 7f1a17c7f700 1 -- 172.21.15.135:0/621346219 wait complete. 2023-11-15T06:43:55.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.035+0000 7f1a17c7f700 1 Processor -- start 2023-11-15T06:43:55.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.035+0000 7f1a17c7f700 1 -- start start 2023-11-15T06:43:55.040 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.035+0000 7f1a17c7f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a10106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:55.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.035+0000 7f1a17c7f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1a10106c20 con 0x7f1a100f5410 2023-11-15T06:43:55.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.035+0000 7f1a15a1b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a10106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:55.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.035+0000 7f1a15a1b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a10106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51812/0 (socket says 172.21.15.135:51812) 2023-11-15T06:43:55.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.035+0000 7f1a15a1b700 1 -- 172.21.15.135:0/1054307174 learned_addr learned my addr 172.21.15.135:0/1054307174 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:55.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.036+0000 7f1a15a1b700 1 -- 172.21.15.135:0/1054307174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1a0c005040 con 0x7f1a100f5410 2023-11-15T06:43:55.041 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.036+0000 7f1a15a1b700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a10106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f1a0c013510 tx=0x7f1a0c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:55.042 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.036+0000 7f1a067fc700 1 -- 172.21.15.135:0/1054307174 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a0c009320 con 0x7f1a100f5410 2023-11-15T06:43:55.042 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.036+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1a10106e20 con 0x7f1a100f5410 2023-11-15T06:43:55.042 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.036+0000 7f1a067fc700 1 -- 172.21.15.135:0/1054307174 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1a0c013f00 con 0x7f1a100f5410 2023-11-15T06:43:55.042 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.036+0000 7f1a067fc700 1 -- 172.21.15.135:0/1054307174 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1a0c014d90 con 0x7f1a100f5410 2023-11-15T06:43:55.042 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.036+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1a101072b0 con 0x7f1a100f5410 2023-11-15T06:43:55.043 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.037+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1a10045ad0 con 0x7f1a100f5410 2023-11-15T06:43:55.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.037+0000 7f1a067fc700 1 -- 172.21.15.135:0/1054307174 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1a0c0095f0 con 0x7f1a100f5410 2023-11-15T06:43:55.044 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.037+0000 7f1a067fc700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19fc02ff60 0x7f19fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:55.046 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.037+0000 7f1a067fc700 1 -- 172.21.15.135:0/1054307174 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1a0c041000 con 0x7f1a100f5410 2023-11-15T06:43:55.046 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.037+0000 7f1a1521a700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19fc02ff60 0x7f19fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:55.046 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.037+0000 7f1a1521a700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19fc02ff60 0x7f19fc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f1a000096a0 tx=0x7f1a00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:55.047 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.041+0000 7f1a067fc700 1 -- 172.21.15.135:0/1054307174 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1a0c014420 con 0x7f1a100f5410 2023-11-15T06:43:55.251 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.248+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1a100fae60 con 0x7f1a100f5410 2023-11-15T06:43:55.252 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.249+0000 7f1a067fc700 1 -- 172.21.15.135:0/1054307174 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1a0c010230 con 0x7f1a100f5410 2023-11-15T06:43:55.252 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:55.252 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:55.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19fc02ff60 msgr2=0x7f19fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:55.253 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19fc02ff60 0x7f19fc032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7f1a000096a0 tx=0x7f1a00006b40).stop 2023-11-15T06:43:55.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 msgr2=0x7f1a10106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:55.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a10106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7f1a0c013510 tx=0x7f1a0c00b040).stop 2023-11-15T06:43:55.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 shutdown_connections 2023-11-15T06:43:55.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f19fc02ff60 0x7f19fc032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:55.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 --2- 172.21.15.135:0/1054307174 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1a100f5410 0x7f1a10106730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:55.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 >> 172.21.15.135:0/1054307174 conn(0x7f1a100f0b70 msgr2=0x7f1a100f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:55.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.250+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 shutdown_connections 2023-11-15T06:43:55.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:55.251+0000 7f1a17c7f700 1 -- 172.21.15.135:0/1054307174 wait complete. 2023-11-15T06:43:55.255 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:55.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:54.288909+0000 mgr.smithi123.razjtn (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:55.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:55.250117+0000 mon.smithi123 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.135:0/1054307174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:56.631 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:56.631 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:57.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.796+0000 7f95ccd57700 1 -- 172.21.15.135:0/661443435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 msgr2=0x7f95c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:57.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.796+0000 7f95ccd57700 1 --2- 172.21.15.135:0/661443435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c80f6930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f95bc009fb0 tx=0x7f95bc009b30).stop 2023-11-15T06:43:57.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.796+0000 7f95ccd57700 1 -- 172.21.15.135:0/661443435 shutdown_connections 2023-11-15T06:43:57.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.796+0000 7f95ccd57700 1 --2- 172.21.15.135:0/661443435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c80f6930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:57.800 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.796+0000 7f95ccd57700 1 -- 172.21.15.135:0/661443435 >> 172.21.15.135:0/661443435 conn(0x7f95c80f1c70 msgr2=0x7f95c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:57.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.796+0000 7f95ccd57700 1 -- 172.21.15.135:0/661443435 shutdown_connections 2023-11-15T06:43:57.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.796+0000 7f95ccd57700 1 -- 172.21.15.135:0/661443435 wait complete. 2023-11-15T06:43:57.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95ccd57700 1 Processor -- start 2023-11-15T06:43:57.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95ccd57700 1 -- start start 2023-11-15T06:43:57.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95ccd57700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:57.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95ccd57700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f95c8106c20 con 0x7f95c80f6510 2023-11-15T06:43:57.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95c659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:57.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95c659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:51832/0 (socket says 172.21.15.135:51832) 2023-11-15T06:43:57.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95c659c700 1 -- 172.21.15.135:0/2182952821 learned_addr learned my addr 172.21.15.135:0/2182952821 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:43:57.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.797+0000 7f95c659c700 1 -- 172.21.15.135:0/2182952821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95bc005040 con 0x7f95c80f6510 2023-11-15T06:43:57.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.798+0000 7f95c659c700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c8106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f95bc013510 tx=0x7f95bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:57.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.798+0000 7f95b6ffd700 1 -- 172.21.15.135:0/2182952821 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc009320 con 0x7f95c80f6510 2023-11-15T06:43:57.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.798+0000 7f95b6ffd700 1 -- 172.21.15.135:0/2182952821 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95bc013f00 con 0x7f95c80f6510 2023-11-15T06:43:57.805 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.798+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f95c8106e20 con 0x7f95c80f6510 2023-11-15T06:43:57.805 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.798+0000 7f95b6ffd700 1 -- 172.21.15.135:0/2182952821 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95bc014d90 con 0x7f95c80f6510 2023-11-15T06:43:57.805 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.798+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f95c81072b0 con 0x7f95c80f6510 2023-11-15T06:43:57.805 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.799+0000 7f95b6ffd700 1 -- 172.21.15.135:0/2182952821 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95bc0095f0 con 0x7f95c80f6510 2023-11-15T06:43:57.805 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.799+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f95c8045ad0 con 0x7f95c80f6510 2023-11-15T06:43:57.806 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.799+0000 7f95b6ffd700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95b002ff60 0x7f95b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:43:57.806 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.799+0000 7f95b6ffd700 1 -- 172.21.15.135:0/2182952821 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95bc040fe0 con 0x7f95c80f6510 2023-11-15T06:43:57.806 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.799+0000 7f95c5d9b700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95b002ff60 0x7f95b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:43:57.806 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.799+0000 7f95c5d9b700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95b002ff60 0x7f95b0032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f95b80096a0 tx=0x7f95b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:43:57.807 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:57.804+0000 7f95b6ffd700 1 -- 172.21.15.135:0/2182952821 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95bc01d7f0 con 0x7f95c80f6510 2023-11-15T06:43:57.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:56.289204+0000 mgr.smithi123.razjtn (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:43:58.011 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.008+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f95c80fbf60 con 0x7f95c80f6510 2023-11-15T06:43:58.012 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.009+0000 7f95b6ffd700 1 -- 172.21.15.135:0/2182952821 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95bc010230 con 0x7f95c80f6510 2023-11-15T06:43:58.012 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:43:58.012 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:43:58.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95b002ff60 msgr2=0x7f95b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:58.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95b002ff60 0x7f95b0032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f95b80096a0 tx=0x7f95b8006b40).stop 2023-11-15T06:43:58.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 msgr2=0x7f95c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:43:58.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c8106730 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f95bc013510 tx=0x7f95bc00b040).stop 2023-11-15T06:43:58.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 shutdown_connections 2023-11-15T06:43:58.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95b002ff60 0x7f95b0032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:58.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 --2- 172.21.15.135:0/2182952821 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f95c80f6510 0x7f95c8106730 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:43:58.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 >> 172.21.15.135:0/2182952821 conn(0x7f95c80f1c70 msgr2=0x7f95c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:43:58.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 shutdown_connections 2023-11-15T06:43:58.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:43:58.011+0000 7f95ccd57700 1 -- 172.21.15.135:0/2182952821 wait complete. 2023-11-15T06:43:58.018 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:43:58.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:43:58.010131+0000 mon.smithi123 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.135:0/2182952821' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:43:59.386 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:43:59.387 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:43:59.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:43:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:43:58.289521+0000 mgr.smithi123.razjtn (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:00.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.524+0000 7f9906531700 1 -- 172.21.15.135:0/936470510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 msgr2=0x7f99000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:00.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.524+0000 7f9906531700 1 --2- 172.21.15.135:0/936470510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000f55c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f98f0009fb0 tx=0x7f98f0009b30).stop 2023-11-15T06:44:00.528 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.524+0000 7f9906531700 1 -- 172.21.15.135:0/936470510 shutdown_connections 2023-11-15T06:44:00.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.524+0000 7f9906531700 1 --2- 172.21.15.135:0/936470510 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000f55c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:00.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.524+0000 7f9906531700 1 -- 172.21.15.135:0/936470510 >> 172.21.15.135:0/936470510 conn(0x7f99000f09a0 msgr2=0x7f99000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:00.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.524+0000 7f9906531700 1 -- 172.21.15.135:0/936470510 shutdown_connections 2023-11-15T06:44:00.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.524+0000 7f9906531700 1 -- 172.21.15.135:0/936470510 wait complete. 2023-11-15T06:44:00.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f9906531700 1 Processor -- start 2023-11-15T06:44:00.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f9906531700 1 -- start start 2023-11-15T06:44:00.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f9906531700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:00.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f9906531700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99000fb5a0 con 0x7f99000f51a0 2023-11-15T06:44:00.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f990552f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:00.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f990552f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:35898/0 (socket says 172.21.15.135:35898) 2023-11-15T06:44:00.531 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f990552f700 1 -- 172.21.15.135:0/3509523405 learned_addr learned my addr 172.21.15.135:0/3509523405 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:00.532 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.525+0000 7f990552f700 1 -- 172.21.15.135:0/3509523405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98f0005040 con 0x7f99000f51a0 2023-11-15T06:44:00.533 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.526+0000 7f990552f700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000fd120 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f98f0000c00 tx=0x7f98f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:00.533 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.526+0000 7f98f5ffb700 1 -- 172.21.15.135:0/3509523405 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98f0009320 con 0x7f99000f51a0 2023-11-15T06:44:00.533 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.526+0000 7f98f5ffb700 1 -- 172.21.15.135:0/3509523405 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98f0013ec0 con 0x7f99000f51a0 2023-11-15T06:44:00.533 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.526+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99000fb740 con 0x7f99000f51a0 2023-11-15T06:44:00.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.526+0000 7f98f5ffb700 1 -- 172.21.15.135:0/3509523405 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98f001ecd0 con 0x7f99000f51a0 2023-11-15T06:44:00.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.526+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99000fbbd0 con 0x7f99000f51a0 2023-11-15T06:44:00.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.527+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9900059100 con 0x7f99000f51a0 2023-11-15T06:44:00.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.527+0000 7f98f5ffb700 1 -- 172.21.15.135:0/3509523405 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98f001e420 con 0x7f99000f51a0 2023-11-15T06:44:00.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.527+0000 7f98f5ffb700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f98ec02ff60 0x7f98ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:00.535 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.527+0000 7f98f5ffb700 1 -- 172.21.15.135:0/3509523405 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98f0040e00 con 0x7f99000f51a0 2023-11-15T06:44:00.535 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.527+0000 7f9904d2e700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f98ec02ff60 0x7f98ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:00.535 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.528+0000 7f9904d2e700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f98ec02ff60 0x7f98ec032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f98fc0096a0 tx=0x7f98fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:00.535 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.532+0000 7f98f5ffb700 1 -- 172.21.15.135:0/3509523405 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98f00136c0 con 0x7f99000f51a0 2023-11-15T06:44:00.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.740+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9900045ad0 con 0x7f99000f51a0 2023-11-15T06:44:00.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.741+0000 7f98f5ffb700 1 -- 172.21.15.135:0/3509523405 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f98f002c2d0 con 0x7f99000f51a0 2023-11-15T06:44:00.744 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:00.744 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:00.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f98ec02ff60 msgr2=0x7f98ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:00.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f98ec02ff60 0x7f98ec032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f98fc0096a0 tx=0x7f98fc006b40).stop 2023-11-15T06:44:00.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 msgr2=0x7f99000fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:00.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000fd120 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f98f0000c00 tx=0x7f98f000b040).stop 2023-11-15T06:44:00.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 shutdown_connections 2023-11-15T06:44:00.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f98ec02ff60 0x7f98ec032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:00.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 --2- 172.21.15.135:0/3509523405 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99000f51a0 0x7f99000fd120 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:00.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 >> 172.21.15.135:0/3509523405 conn(0x7f99000f09a0 msgr2=0x7f990005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:00.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.742+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 shutdown_connections 2023-11-15T06:44:00.747 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:00.743+0000 7f9906531700 1 -- 172.21.15.135:0/3509523405 wait complete. 2023-11-15T06:44:00.748 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:01.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:00.289815+0000 mgr.smithi123.razjtn (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:01.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:00.741937+0000 mon.smithi123 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.135:0/3509523405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:02.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:02.124 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:03.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2325834536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:03.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 --2- 172.21.15.135:0/2325834536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f6a60009fb0 tx=0x7f6a60009b30).stop 2023-11-15T06:44:03.302 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2325834536 shutdown_connections 2023-11-15T06:44:03.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 --2- 172.21.15.135:0/2325834536 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700f5830 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:03.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2325834536 >> 172.21.15.135:0/2325834536 conn(0x7f6a700f0b70 msgr2=0x7f6a700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:03.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2325834536 shutdown_connections 2023-11-15T06:44:03.303 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2325834536 wait complete. 2023-11-15T06:44:03.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 Processor -- start 2023-11-15T06:44:03.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 -- start start 2023-11-15T06:44:03.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:03.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a75eb5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a700fb820 con 0x7f6a700f5410 2023-11-15T06:44:03.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a6f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:03.304 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.298+0000 7f6a6f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:35910/0 (socket says 172.21.15.135:35910) 2023-11-15T06:44:03.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a6f7fe700 1 -- 172.21.15.135:0/2938060636 learned_addr learned my addr 172.21.15.135:0/2938060636 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:03.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a6f7fe700 1 -- 172.21.15.135:0/2938060636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a60005040 con 0x7f6a700f5410 2023-11-15T06:44:03.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a6f7fe700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f6a60013510 tx=0x7f6a6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:03.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a6cff9700 1 -- 172.21.15.135:0/2938060636 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a60009320 con 0x7f6a700f5410 2023-11-15T06:44:03.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a700fb9c0 con 0x7f6a700f5410 2023-11-15T06:44:03.306 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a6cff9700 1 -- 172.21.15.135:0/2938060636 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a60013f00 con 0x7f6a700f5410 2023-11-15T06:44:03.306 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a6cff9700 1 -- 172.21.15.135:0/2938060636 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a60014d90 con 0x7f6a700f5410 2023-11-15T06:44:03.306 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.299+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a700fbe50 con 0x7f6a700f5410 2023-11-15T06:44:03.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.300+0000 7f6a6cff9700 1 -- 172.21.15.135:0/2938060636 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a60014420 con 0x7f6a700f5410 2023-11-15T06:44:03.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.300+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a70059100 con 0x7f6a700f5410 2023-11-15T06:44:03.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.300+0000 7f6a6cff9700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:03.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.300+0000 7f6a6cff9700 1 -- 172.21.15.135:0/2938060636 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a600410e0 con 0x7f6a700f5410 2023-11-15T06:44:03.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.300+0000 7f6a67fff700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:03.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.301+0000 7f6a67fff700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6a5c02ff60 0x7f6a5c032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6a580096a0 tx=0x7f6a58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:03.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.305+0000 7f6a6cff9700 1 -- 172.21.15.135:0/2938060636 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a600095f0 con 0x7f6a700f5410 2023-11-15T06:44:03.515 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.513+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a70045ad0 con 0x7f6a700f5410 2023-11-15T06:44:03.516 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.513+0000 7f6a6cff9700 1 -- 172.21.15.135:0/2938060636 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a6001db70 con 0x7f6a700f5410 2023-11-15T06:44:03.516 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:03.517 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:03.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6a5c02ff60 msgr2=0x7f6a5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:03.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6a5c02ff60 0x7f6a5c032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f6a580096a0 tx=0x7f6a58006b40).stop 2023-11-15T06:44:03.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 msgr2=0x7f6a700fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:03.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f6a60013510 tx=0x7f6a6000b040).stop 2023-11-15T06:44:03.518 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 shutdown_connections 2023-11-15T06:44:03.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6a5c02ff60 0x7f6a5c032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:03.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 --2- 172.21.15.135:0/2938060636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6a700f5410 0x7f6a700fd3a0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:03.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 >> 172.21.15.135:0/2938060636 conn(0x7f6a700f0b70 msgr2=0x7f6a7005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:03.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 shutdown_connections 2023-11-15T06:44:03.519 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:03.515+0000 7f6a75eb5700 1 -- 172.21.15.135:0/2938060636 wait complete. 2023-11-15T06:44:03.520 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:02.290142+0000 mgr.smithi123.razjtn (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:03.514338+0000 mon.smithi123 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.135:0/2938060636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:04.897 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:04.897 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:05.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:04.290486+0000 mgr.smithi123.razjtn (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:06.017 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 -- 172.21.15.135:0/2285083393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 msgr2=0x7f28c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:06.017 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 --2- 172.21.15.135:0/2285083393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c80f5830 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7f28bc009fb0 tx=0x7f28bc009b30).stop 2023-11-15T06:44:06.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 -- 172.21.15.135:0/2285083393 shutdown_connections 2023-11-15T06:44:06.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 --2- 172.21.15.135:0/2285083393 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c80f5830 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:06.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 -- 172.21.15.135:0/2285083393 >> 172.21.15.135:0/2285083393 conn(0x7f28c80f0b70 msgr2=0x7f28c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:06.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 -- 172.21.15.135:0/2285083393 shutdown_connections 2023-11-15T06:44:06.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 -- 172.21.15.135:0/2285083393 wait complete. 2023-11-15T06:44:06.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 Processor -- start 2023-11-15T06:44:06.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.011+0000 7f28cf045700 1 -- start start 2023-11-15T06:44:06.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.012+0000 7f28cf045700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:06.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.012+0000 7f28cf045700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f28c8105b20 con 0x7f28c80f5410 2023-11-15T06:44:06.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.012+0000 7f28ccde1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:06.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.012+0000 7f28ccde1700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:35926/0 (socket says 172.21.15.135:35926) 2023-11-15T06:44:06.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.012+0000 7f28ccde1700 1 -- 172.21.15.135:0/1834448218 learned_addr learned my addr 172.21.15.135:0/1834448218 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:06.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.012+0000 7f28ccde1700 1 -- 172.21.15.135:0/1834448218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f28bc005040 con 0x7f28c80f5410 2023-11-15T06:44:06.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.013+0000 7f28ccde1700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c8105630 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f28bc013510 tx=0x7f28bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:06.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.013+0000 7f28c57fa700 1 -- 172.21.15.135:0/1834448218 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28bc009320 con 0x7f28c80f5410 2023-11-15T06:44:06.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.013+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f28c8105d20 con 0x7f28c80f5410 2023-11-15T06:44:06.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.013+0000 7f28c57fa700 1 -- 172.21.15.135:0/1834448218 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f28bc013f00 con 0x7f28c80f5410 2023-11-15T06:44:06.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.013+0000 7f28c57fa700 1 -- 172.21.15.135:0/1834448218 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f28bc014d90 con 0x7f28c80f5410 2023-11-15T06:44:06.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.013+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28c81061b0 con 0x7f28c80f5410 2023-11-15T06:44:06.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.013+0000 7f28c57fa700 1 -- 172.21.15.135:0/1834448218 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28bc0263a0 con 0x7f28c80f5410 2023-11-15T06:44:06.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.014+0000 7f28c57fa700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f28b802ff60 0x7f28b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:06.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.014+0000 7f28c57fa700 1 -- 172.21.15.135:0/1834448218 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28bc041010 con 0x7f28c80f5410 2023-11-15T06:44:06.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.014+0000 7f28c7fff700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f28b802ff60 0x7f28b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:06.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.014+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f28c8059100 con 0x7f28c80f5410 2023-11-15T06:44:06.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.014+0000 7f28c7fff700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f28b802ff60 0x7f28b8032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f28b40096a0 tx=0x7f28b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:06.024 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.019+0000 7f28c57fa700 1 -- 172.21.15.135:0/1834448218 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28bc012920 con 0x7f28c80f5410 2023-11-15T06:44:06.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.228+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28c80fae60 con 0x7f28c80f5410 2023-11-15T06:44:06.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.228+0000 7f28c57fa700 1 -- 172.21.15.135:0/1834448218 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f28bc02b020 con 0x7f28c80f5410 2023-11-15T06:44:06.232 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:06.232 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:06.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f28b802ff60 msgr2=0x7f28b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:06.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f28b802ff60 0x7f28b8032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7f28b40096a0 tx=0x7f28b4006b40).stop 2023-11-15T06:44:06.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 msgr2=0x7f28c8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:06.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c8105630 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7f28bc013510 tx=0x7f28bc00b040).stop 2023-11-15T06:44:06.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 shutdown_connections 2023-11-15T06:44:06.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f28b802ff60 0x7f28b8032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:06.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 --2- 172.21.15.135:0/1834448218 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f28c80f5410 0x7f28c8105630 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:06.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 >> 172.21.15.135:0/1834448218 conn(0x7f28c80f0b70 msgr2=0x7f28c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:06.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 shutdown_connections 2023-11-15T06:44:06.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:06.230+0000 7f28cf045700 1 -- 172.21.15.135:0/1834448218 wait complete. 2023-11-15T06:44:06.236 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:06.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:06 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:06.229539+0000 mon.smithi123 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.135:0/1834448218' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:07.610 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:07.611 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:07.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:06.290775+0000 mgr.smithi123.razjtn (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:08.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.749+0000 7fb493bf3700 1 -- 172.21.15.135:0/3132674120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 msgr2=0x7fb48c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:08.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.749+0000 7fb493bf3700 1 --2- 172.21.15.135:0/3132674120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c0f5830 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fb488009fb0 tx=0x7fb488009b30).stop 2023-11-15T06:44:08.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.749+0000 7fb493bf3700 1 -- 172.21.15.135:0/3132674120 shutdown_connections 2023-11-15T06:44:08.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.749+0000 7fb493bf3700 1 --2- 172.21.15.135:0/3132674120 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c0f5830 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:08.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.749+0000 7fb493bf3700 1 -- 172.21.15.135:0/3132674120 >> 172.21.15.135:0/3132674120 conn(0x7fb48c0f0b70 msgr2=0x7fb48c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:08.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.749+0000 7fb493bf3700 1 -- 172.21.15.135:0/3132674120 shutdown_connections 2023-11-15T06:44:08.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.749+0000 7fb493bf3700 1 -- 172.21.15.135:0/3132674120 wait complete. 2023-11-15T06:44:08.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb493bf3700 1 Processor -- start 2023-11-15T06:44:08.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb493bf3700 1 -- start start 2023-11-15T06:44:08.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb493bf3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:08.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb493bf3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb48c105b20 con 0x7fb48c0f5410 2023-11-15T06:44:08.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb49198f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:08.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb49198f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:46954/0 (socket says 172.21.15.135:46954) 2023-11-15T06:44:08.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb49198f700 1 -- 172.21.15.135:0/565714924 learned_addr learned my addr 172.21.15.135:0/565714924 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:08.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb49198f700 1 -- 172.21.15.135:0/565714924 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb488005040 con 0x7fb48c0f5410 2023-11-15T06:44:08.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.750+0000 7fb49198f700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c105630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb488013510 tx=0x7fb48800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:08.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.751+0000 7fb4827fc700 1 -- 172.21.15.135:0/565714924 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb488009320 con 0x7fb48c0f5410 2023-11-15T06:44:08.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.751+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb48c105d20 con 0x7fb48c0f5410 2023-11-15T06:44:08.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.751+0000 7fb4827fc700 1 -- 172.21.15.135:0/565714924 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb488013f00 con 0x7fb48c0f5410 2023-11-15T06:44:08.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.751+0000 7fb4827fc700 1 -- 172.21.15.135:0/565714924 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb488014d90 con 0x7fb48c0f5410 2023-11-15T06:44:08.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.751+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb48c1061b0 con 0x7fb48c0f5410 2023-11-15T06:44:08.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.751+0000 7fb4827fc700 1 -- 172.21.15.135:0/565714924 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb4880263a0 con 0x7fb48c0f5410 2023-11-15T06:44:08.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.751+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb48c045ad0 con 0x7fb48c0f5410 2023-11-15T06:44:08.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.752+0000 7fb4827fc700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb47802ffb0 0x7fb478032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:08.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.752+0000 7fb4827fc700 1 -- 172.21.15.135:0/565714924 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb4880412b0 con 0x7fb48c0f5410 2023-11-15T06:44:08.759 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.752+0000 7fb49118e700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb47802ffb0 0x7fb478032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:08.760 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.752+0000 7fb49118e700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb47802ffb0 0x7fb478032470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb47c0096a0 tx=0x7fb47c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:08.760 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.756+0000 7fb4827fc700 1 -- 172.21.15.135:0/565714924 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb4880126f0 con 0x7fb48c0f5410 2023-11-15T06:44:08.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.963+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb48c0fae60 con 0x7fb48c0f5410 2023-11-15T06:44:08.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.964+0000 7fb4827fc700 1 -- 172.21.15.135:0/565714924 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb488012330 con 0x7fb48c0f5410 2023-11-15T06:44:08.967 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:08.967 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:08.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.965+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb47802ffb0 msgr2=0x7fb478032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:08.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.965+0000 7fb493bf3700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb47802ffb0 0x7fb478032470 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fb47c0096a0 tx=0x7fb47c006b40).stop 2023-11-15T06:44:08.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.965+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 msgr2=0x7fb48c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:08.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.965+0000 7fb493bf3700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c105630 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fb488013510 tx=0x7fb48800b040).stop 2023-11-15T06:44:08.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.966+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 shutdown_connections 2023-11-15T06:44:08.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.966+0000 7fb493bf3700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb47802ffb0 0x7fb478032470 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:08.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.966+0000 7fb493bf3700 1 --2- 172.21.15.135:0/565714924 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb48c0f5410 0x7fb48c105630 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:08.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.966+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 >> 172.21.15.135:0/565714924 conn(0x7fb48c0f0b70 msgr2=0x7fb48c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:08.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.966+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 shutdown_connections 2023-11-15T06:44:08.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:08.966+0000 7fb493bf3700 1 -- 172.21.15.135:0/565714924 wait complete. 2023-11-15T06:44:08.971 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:08.291054+0000 mgr.smithi123.razjtn (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:08.965124+0000 mon.smithi123 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.135:0/565714924' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:10.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:10.390 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:11.524 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 -- 172.21.15.135:0/4165203681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 msgr2=0x7f96180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:11.524 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 --2- 172.21.15.135:0/4165203681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96180f66c0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f9608009fb0 tx=0x7f9608009b30).stop 2023-11-15T06:44:11.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 -- 172.21.15.135:0/4165203681 shutdown_connections 2023-11-15T06:44:11.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 --2- 172.21.15.135:0/4165203681 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96180f66c0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:11.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 -- 172.21.15.135:0/4165203681 >> 172.21.15.135:0/4165203681 conn(0x7f96180f1aa0 msgr2=0x7f96180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:11.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 -- 172.21.15.135:0/4165203681 shutdown_connections 2023-11-15T06:44:11.525 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 -- 172.21.15.135:0/4165203681 wait complete. 2023-11-15T06:44:11.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 Processor -- start 2023-11-15T06:44:11.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.520+0000 7f961e429700 1 -- start start 2023-11-15T06:44:11.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.521+0000 7f961e429700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96181064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:11.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.521+0000 7f961e429700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f96181069a0 con 0x7f96180f62a0 2023-11-15T06:44:11.526 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.521+0000 7f961d427700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96181064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:11.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.521+0000 7f961d427700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96181064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:46978/0 (socket says 172.21.15.135:46978) 2023-11-15T06:44:11.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.521+0000 7f961d427700 1 -- 172.21.15.135:0/2411897802 learned_addr learned my addr 172.21.15.135:0/2411897802 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:11.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.521+0000 7f961d427700 1 -- 172.21.15.135:0/2411897802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9608005040 con 0x7f96180f62a0 2023-11-15T06:44:11.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.521+0000 7f961d427700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96181064b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f9608013510 tx=0x7f960800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:11.533 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.522+0000 7f960dffb700 1 -- 172.21.15.135:0/2411897802 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9608009320 con 0x7f96180f62a0 2023-11-15T06:44:11.533 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.522+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9618106ba0 con 0x7f96180f62a0 2023-11-15T06:44:11.533 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.522+0000 7f960dffb700 1 -- 172.21.15.135:0/2411897802 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9608013f00 con 0x7f96180f62a0 2023-11-15T06:44:11.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.522+0000 7f960dffb700 1 -- 172.21.15.135:0/2411897802 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9608014dc0 con 0x7f96180f62a0 2023-11-15T06:44:11.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.522+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9618107030 con 0x7f96180f62a0 2023-11-15T06:44:11.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.523+0000 7f960dffb700 1 -- 172.21.15.135:0/2411897802 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f96080095f0 con 0x7f96180f62a0 2023-11-15T06:44:11.534 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.523+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9618045ad0 con 0x7f96180f62a0 2023-11-15T06:44:11.535 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.523+0000 7f960dffb700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f960402ff60 0x7f9604032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:11.535 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.523+0000 7f960dffb700 1 -- 172.21.15.135:0/2411897802 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9608041100 con 0x7f96180f62a0 2023-11-15T06:44:11.535 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.523+0000 7f961cc26700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f960402ff60 0x7f9604032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:11.536 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.524+0000 7f961cc26700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f960402ff60 0x7f9604032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f96140096a0 tx=0x7f9614006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:11.536 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.528+0000 7f960dffb700 1 -- 172.21.15.135:0/2411897802 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9608014420 con 0x7f96180f62a0 2023-11-15T06:44:11.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.738+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9618107ee0 con 0x7f96180f62a0 2023-11-15T06:44:11.742 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.739+0000 7f960dffb700 1 -- 172.21.15.135:0/2411897802 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9608010230 con 0x7f96180f62a0 2023-11-15T06:44:11.742 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:11.742 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:11.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f960402ff60 msgr2=0x7f9604032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:11.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f960402ff60 0x7f9604032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f96140096a0 tx=0x7f9614006b40).stop 2023-11-15T06:44:11.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 msgr2=0x7f96181064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:11.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96181064b0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f9608013510 tx=0x7f960800b040).stop 2023-11-15T06:44:11.745 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 shutdown_connections 2023-11-15T06:44:11.745 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f960402ff60 0x7f9604032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:11.745 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 --2- 172.21.15.135:0/2411897802 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f96180f62a0 0x7f96181064b0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:11.745 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 >> 172.21.15.135:0/2411897802 conn(0x7f96180f1aa0 msgr2=0x7f96180f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:11.745 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 shutdown_connections 2023-11-15T06:44:11.746 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:11.741+0000 7f961e429700 1 -- 172.21.15.135:0/2411897802 wait complete. 2023-11-15T06:44:11.746 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:11.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:10.291465+0000 mgr.smithi123.razjtn (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:12.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:12 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:11.740096+0000 mon.smithi123 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.135:0/2411897802' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:13.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:13.149 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:13.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:12.291845+0000 mgr.smithi123.razjtn (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:14.314 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.309+0000 7f339a10a700 1 -- 172.21.15.135:0/3809899811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 msgr2=0x7f33940f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:14.314 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.309+0000 7f339a10a700 1 --2- 172.21.15.135:0/3809899811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33940f55c0 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f3384009fb0 tx=0x7f3384009b30).stop 2023-11-15T06:44:14.314 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.309+0000 7f339a10a700 1 -- 172.21.15.135:0/3809899811 shutdown_connections 2023-11-15T06:44:14.315 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.309+0000 7f339a10a700 1 --2- 172.21.15.135:0/3809899811 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33940f55c0 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:14.315 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.309+0000 7f339a10a700 1 -- 172.21.15.135:0/3809899811 >> 172.21.15.135:0/3809899811 conn(0x7f33940f09a0 msgr2=0x7f33940f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:14.315 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f339a10a700 1 -- 172.21.15.135:0/3809899811 shutdown_connections 2023-11-15T06:44:14.315 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f339a10a700 1 -- 172.21.15.135:0/3809899811 wait complete. 2023-11-15T06:44:14.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f339a10a700 1 Processor -- start 2023-11-15T06:44:14.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f339a10a700 1 -- start start 2023-11-15T06:44:14.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f339a10a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33941064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:14.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f339a10a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33941069a0 con 0x7f33940f51a0 2023-11-15T06:44:14.316 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f3399108700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33941064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:14.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f3399108700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33941064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:46990/0 (socket says 172.21.15.135:46990) 2023-11-15T06:44:14.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.310+0000 7f3399108700 1 -- 172.21.15.135:0/4149397859 learned_addr learned my addr 172.21.15.135:0/4149397859 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:14.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.311+0000 7f3399108700 1 -- 172.21.15.135:0/4149397859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3384005040 con 0x7f33940f51a0 2023-11-15T06:44:14.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.311+0000 7f3399108700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33941064b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f3384013510 tx=0x7f338400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:14.317 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.311+0000 7f3389ffb700 1 -- 172.21.15.135:0/4149397859 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3384009320 con 0x7f33940f51a0 2023-11-15T06:44:14.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.311+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3394106ba0 con 0x7f33940f51a0 2023-11-15T06:44:14.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.311+0000 7f3389ffb700 1 -- 172.21.15.135:0/4149397859 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3384013f00 con 0x7f33940f51a0 2023-11-15T06:44:14.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.311+0000 7f3389ffb700 1 -- 172.21.15.135:0/4149397859 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3384014dc0 con 0x7f33940f51a0 2023-11-15T06:44:14.318 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.311+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3394107030 con 0x7f33940f51a0 2023-11-15T06:44:14.319 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.312+0000 7f3389ffb700 1 -- 172.21.15.135:0/4149397859 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f338401d3d0 con 0x7f33940f51a0 2023-11-15T06:44:14.320 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.312+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3394045ad0 con 0x7f33940f51a0 2023-11-15T06:44:14.320 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.312+0000 7f3389ffb700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f338002ff60 0x7f3380032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:14.322 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.312+0000 7f3389ffb700 1 -- 172.21.15.135:0/4149397859 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f33840413c0 con 0x7f33940f51a0 2023-11-15T06:44:14.322 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.313+0000 7f3398907700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f338002ff60 0x7f3380032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:14.322 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.313+0000 7f3398907700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f338002ff60 0x7f3380032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f33900096a0 tx=0x7f3390006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:14.322 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.317+0000 7f3389ffb700 1 -- 172.21.15.135:0/4149397859 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33840126f0 con 0x7f33940f51a0 2023-11-15T06:44:14.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.524+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3394107ee0 con 0x7f33940f51a0 2023-11-15T06:44:14.527 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.525+0000 7f3389ffb700 1 -- 172.21.15.135:0/4149397859 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3384012330 con 0x7f33940f51a0 2023-11-15T06:44:14.528 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:14.528 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:14.529 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.526+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f338002ff60 msgr2=0x7f3380032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:14.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.526+0000 7f339a10a700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f338002ff60 0x7f3380032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f33900096a0 tx=0x7f3390006b40).stop 2023-11-15T06:44:14.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 msgr2=0x7f33941064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:14.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33941064b0 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f3384013510 tx=0x7f338400b040).stop 2023-11-15T06:44:14.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 shutdown_connections 2023-11-15T06:44:14.530 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f338002ff60 0x7f3380032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:14.531 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 --2- 172.21.15.135:0/4149397859 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f33940f51a0 0x7f33941064b0 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:14.531 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 >> 172.21.15.135:0/4149397859 conn(0x7f33940f09a0 msgr2=0x7f339405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:14.531 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 shutdown_connections 2023-11-15T06:44:14.531 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:14.527+0000 7f339a10a700 1 -- 172.21.15.135:0/4149397859 wait complete. 2023-11-15T06:44:14.532 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:14.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:14 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:14.525861+0000 mon.smithi123 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.135:0/4149397859' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:15.877 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:15.878 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:15.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:14.292110+0000 mgr.smithi123.razjtn (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:17.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.010+0000 7f50faf9e700 1 -- 172.21.15.135:0/3024886590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 msgr2=0x7f50f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:17.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.010+0000 7f50faf9e700 1 --2- 172.21.15.135:0/3024886590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f40f5830 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f50e8009fb0 tx=0x7f50e8009b30).stop 2023-11-15T06:44:17.014 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.010+0000 7f50faf9e700 1 -- 172.21.15.135:0/3024886590 shutdown_connections 2023-11-15T06:44:17.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.010+0000 7f50faf9e700 1 --2- 172.21.15.135:0/3024886590 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f40f5830 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:17.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.010+0000 7f50faf9e700 1 -- 172.21.15.135:0/3024886590 >> 172.21.15.135:0/3024886590 conn(0x7f50f40f0b70 msgr2=0x7f50f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:17.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.010+0000 7f50faf9e700 1 -- 172.21.15.135:0/3024886590 shutdown_connections 2023-11-15T06:44:17.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.010+0000 7f50faf9e700 1 -- 172.21.15.135:0/3024886590 wait complete. 2023-11-15T06:44:17.015 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.011+0000 7f50faf9e700 1 Processor -- start 2023-11-15T06:44:17.016 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.011+0000 7f50faf9e700 1 -- start start 2023-11-15T06:44:17.017 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.011+0000 7f50faf9e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:17.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.011+0000 7f50faf9e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f50f4060240 con 0x7f50f40f5410 2023-11-15T06:44:17.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f8d3a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:17.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f8d3a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:47010/0 (socket says 172.21.15.135:47010) 2023-11-15T06:44:17.018 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f8d3a700 1 -- 172.21.15.135:0/3094771482 learned_addr learned my addr 172.21.15.135:0/3094771482 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:17.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f8d3a700 1 -- 172.21.15.135:0/3094771482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f50e8005040 con 0x7f50f40f5410 2023-11-15T06:44:17.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f8d3a700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f4109430 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f50e8013510 tx=0x7f50e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:17.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f17fa700 1 -- 172.21.15.135:0/3094771482 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e8009320 con 0x7f50f40f5410 2023-11-15T06:44:17.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f50f4060440 con 0x7f50f40f5410 2023-11-15T06:44:17.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f17fa700 1 -- 172.21.15.135:0/3094771482 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f50e8013f00 con 0x7f50f40f5410 2023-11-15T06:44:17.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50f17fa700 1 -- 172.21.15.135:0/3094771482 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f50e8014d90 con 0x7f50f40f5410 2023-11-15T06:44:17.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.012+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f50f4109ae0 con 0x7f50f40f5410 2023-11-15T06:44:17.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.013+0000 7f50f17fa700 1 -- 172.21.15.135:0/3094771482 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f50e8014420 con 0x7f50f40f5410 2023-11-15T06:44:17.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.013+0000 7f50f17fa700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f50e402ff60 0x7f50e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:17.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.013+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f50f4045ad0 con 0x7f50f40f5410 2023-11-15T06:44:17.024 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.013+0000 7f50f17fa700 1 -- 172.21.15.135:0/3094771482 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f50e80410c0 con 0x7f50f40f5410 2023-11-15T06:44:17.024 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.014+0000 7f50f3fff700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f50e402ff60 0x7f50e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:17.024 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.014+0000 7f50f3fff700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f50e402ff60 0x7f50e4032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f50e00096a0 tx=0x7f50e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:17.025 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.018+0000 7f50f17fa700 1 -- 172.21.15.135:0/3094771482 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f50e806d2b0 con 0x7f50f40f5410 2023-11-15T06:44:17.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.224+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50f40faf50 con 0x7f50f40f5410 2023-11-15T06:44:17.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.225+0000 7f50f17fa700 1 -- 172.21.15.135:0/3094771482 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f50e801db70 con 0x7f50f40f5410 2023-11-15T06:44:17.228 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:17.228 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:17.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f50e402ff60 msgr2=0x7f50e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:17.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f50e402ff60 0x7f50e4032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f50e00096a0 tx=0x7f50e0006b40).stop 2023-11-15T06:44:17.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 msgr2=0x7f50f4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:17.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f4109430 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f50e8013510 tx=0x7f50e800b040).stop 2023-11-15T06:44:17.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 shutdown_connections 2023-11-15T06:44:17.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f50e402ff60 0x7f50e4032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:17.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 --2- 172.21.15.135:0/3094771482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f50f40f5410 0x7f50f4109430 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:17.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 >> 172.21.15.135:0/3094771482 conn(0x7f50f40f0b70 msgr2=0x7f50f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:17.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 shutdown_connections 2023-11-15T06:44:17.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:17.227+0000 7f50faf9e700 1 -- 172.21.15.135:0/3094771482 wait complete. 2023-11-15T06:44:17.232 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:17.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:16.292477+0000 mgr.smithi123.razjtn (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:17.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:17.226206+0000 mon.smithi123 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.135:0/3094771482' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:18.625 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:18.626 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:19.776 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 -- 172.21.15.135:0/32509113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 msgr2=0x7ffa500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:19.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 --2- 172.21.15.135:0/32509113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa500f55c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ffa40009fb0 tx=0x7ffa40009b30).stop 2023-11-15T06:44:19.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 -- 172.21.15.135:0/32509113 shutdown_connections 2023-11-15T06:44:19.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 --2- 172.21.15.135:0/32509113 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa500f55c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:19.777 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 -- 172.21.15.135:0/32509113 >> 172.21.15.135:0/32509113 conn(0x7ffa500f09a0 msgr2=0x7ffa500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:19.778 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 -- 172.21.15.135:0/32509113 shutdown_connections 2023-11-15T06:44:19.778 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 -- 172.21.15.135:0/32509113 wait complete. 2023-11-15T06:44:19.778 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 Processor -- start 2023-11-15T06:44:19.778 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 -- start start 2023-11-15T06:44:19.778 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa501053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:19.779 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.772+0000 7ffa54b7f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffa501058a0 con 0x7ffa500f51a0 2023-11-15T06:44:19.779 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa4f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa501053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:19.779 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa4f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa501053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33194/0 (socket says 172.21.15.135:33194) 2023-11-15T06:44:19.779 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa4f7fe700 1 -- 172.21.15.135:0/2773322874 learned_addr learned my addr 172.21.15.135:0/2773322874 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:19.779 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa4f7fe700 1 -- 172.21.15.135:0/2773322874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffa40005040 con 0x7ffa500f51a0 2023-11-15T06:44:19.780 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa4f7fe700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa501053b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ffa40009830 tx=0x7ffa4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:19.780 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa37fff700 1 -- 172.21.15.135:0/2773322874 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa40009320 con 0x7ffa500f51a0 2023-11-15T06:44:19.780 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffa50105aa0 con 0x7ffa500f51a0 2023-11-15T06:44:19.780 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa37fff700 1 -- 172.21.15.135:0/2773322874 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffa400136c0 con 0x7ffa500f51a0 2023-11-15T06:44:19.780 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa37fff700 1 -- 172.21.15.135:0/2773322874 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffa4001ee90 con 0x7ffa500f51a0 2023-11-15T06:44:19.781 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.773+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffa50105f30 con 0x7ffa500f51a0 2023-11-15T06:44:19.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.774+0000 7ffa37fff700 1 -- 172.21.15.135:0/2773322874 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffa4001e420 con 0x7ffa500f51a0 2023-11-15T06:44:19.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.774+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffa50045ad0 con 0x7ffa500f51a0 2023-11-15T06:44:19.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.774+0000 7ffa37fff700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffa3802ff60 0x7ffa38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:19.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.774+0000 7ffa37fff700 1 -- 172.21.15.135:0/2773322874 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffa40040fa0 con 0x7ffa500f51a0 2023-11-15T06:44:19.782 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.774+0000 7ffa4effd700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffa3802ff60 0x7ffa38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:19.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.775+0000 7ffa4effd700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffa3802ff60 0x7ffa38032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ffa440096a0 tx=0x7ffa44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:19.783 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.779+0000 7ffa37fff700 1 -- 172.21.15.135:0/2773322874 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffa400095f0 con 0x7ffa500f51a0 2023-11-15T06:44:19.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:18.292843+0000 mgr.smithi123.razjtn (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:19.990 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.987+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffa50106de0 con 0x7ffa500f51a0 2023-11-15T06:44:19.991 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.988+0000 7ffa37fff700 1 -- 172.21.15.135:0/2773322874 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffa4002c2d0 con 0x7ffa500f51a0 2023-11-15T06:44:19.991 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:19.991 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:19.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffa3802ff60 msgr2=0x7ffa38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:19.992 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffa3802ff60 0x7ffa38032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ffa440096a0 tx=0x7ffa44006b40).stop 2023-11-15T06:44:19.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 msgr2=0x7ffa501053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:19.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa501053b0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ffa40009830 tx=0x7ffa4000b040).stop 2023-11-15T06:44:19.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 shutdown_connections 2023-11-15T06:44:19.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffa3802ff60 0x7ffa38032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:19.993 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 --2- 172.21.15.135:0/2773322874 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffa500f51a0 0x7ffa501053b0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:19.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 >> 172.21.15.135:0/2773322874 conn(0x7ffa500f09a0 msgr2=0x7ffa500f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:19.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 shutdown_connections 2023-11-15T06:44:19.994 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:19.989+0000 7ffa54b7f700 1 -- 172.21.15.135:0/2773322874 wait complete. 2023-11-15T06:44:19.995 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:20.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:20 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:19.988994+0000 mon.smithi123 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.135:0/2773322874' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:21.318 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:21.319 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:21.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:20.293058+0000 mgr.smithi123.razjtn (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:22.476 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.472+0000 7fee67fff700 1 -- 172.21.15.135:0/1878708727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 msgr2=0x7fee680f8f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:22.477 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.472+0000 7fee67fff700 1 --2- 172.21.15.135:0/1878708727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f8f80 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7fee58009fb0 tx=0x7fee58009b30).stop 2023-11-15T06:44:22.477 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.472+0000 7fee67fff700 1 -- 172.21.15.135:0/1878708727 shutdown_connections 2023-11-15T06:44:22.477 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.472+0000 7fee67fff700 1 --2- 172.21.15.135:0/1878708727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f8f80 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:22.478 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.472+0000 7fee67fff700 1 -- 172.21.15.135:0/1878708727 >> 172.21.15.135:0/1878708727 conn(0x7fee680f09a0 msgr2=0x7fee680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:22.478 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.472+0000 7fee67fff700 1 -- 172.21.15.135:0/1878708727 shutdown_connections 2023-11-15T06:44:22.478 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.473+0000 7fee67fff700 1 -- 172.21.15.135:0/1878708727 wait complete. 2023-11-15T06:44:22.478 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.473+0000 7fee67fff700 1 Processor -- start 2023-11-15T06:44:22.479 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.473+0000 7fee67fff700 1 -- start start 2023-11-15T06:44:22.479 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.473+0000 7fee67fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f6a00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:22.479 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.473+0000 7fee67fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fee680f4e80 con 0x7fee68060180 2023-11-15T06:44:22.479 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee66ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f6a00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:22.479 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee66ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f6a00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33216/0 (socket says 172.21.15.135:33216) 2023-11-15T06:44:22.480 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee66ffd700 1 -- 172.21.15.135:0/2994482668 learned_addr learned my addr 172.21.15.135:0/2994482668 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:22.480 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee66ffd700 1 -- 172.21.15.135:0/2994482668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fee58005040 con 0x7fee68060180 2023-11-15T06:44:22.481 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee66ffd700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f6a00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fee58013510 tx=0x7fee5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:22.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee4f7fe700 1 -- 172.21.15.135:0/2994482668 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee58009320 con 0x7fee68060180 2023-11-15T06:44:22.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fee680f5020 con 0x7fee68060180 2023-11-15T06:44:22.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee4f7fe700 1 -- 172.21.15.135:0/2994482668 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fee58013f00 con 0x7fee68060180 2023-11-15T06:44:22.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee4f7fe700 1 -- 172.21.15.135:0/2994482668 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fee58014d90 con 0x7fee68060180 2023-11-15T06:44:22.482 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.474+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fee680f54b0 con 0x7fee68060180 2023-11-15T06:44:22.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.475+0000 7fee4f7fe700 1 -- 172.21.15.135:0/2994482668 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fee580095f0 con 0x7fee68060180 2023-11-15T06:44:22.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.475+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fee68059100 con 0x7fee68060180 2023-11-15T06:44:22.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.475+0000 7fee4f7fe700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee5002ff60 0x7fee50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:22.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.475+0000 7fee4f7fe700 1 -- 172.21.15.135:0/2994482668 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fee580410c0 con 0x7fee68060180 2023-11-15T06:44:22.483 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.475+0000 7fee667fc700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee5002ff60 0x7fee50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:22.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.476+0000 7fee667fc700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee5002ff60 0x7fee50032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fee5c0096a0 tx=0x7fee5c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:22.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.480+0000 7fee4f7fe700 1 -- 172.21.15.135:0/2994482668 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fee5801d7f0 con 0x7fee68060180 2023-11-15T06:44:22.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.684+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fee680fbce0 con 0x7fee68060180 2023-11-15T06:44:22.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.684+0000 7fee4f7fe700 1 -- 172.21.15.135:0/2994482668 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fee58010230 con 0x7fee68060180 2023-11-15T06:44:22.688 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:22.688 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:22.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee5002ff60 msgr2=0x7fee50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:22.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee5002ff60 0x7fee50032420 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fee5c0096a0 tx=0x7fee5c006b40).stop 2023-11-15T06:44:22.689 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 msgr2=0x7fee680f6a00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:22.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f6a00 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7fee58013510 tx=0x7fee5800b040).stop 2023-11-15T06:44:22.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 shutdown_connections 2023-11-15T06:44:22.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fee5002ff60 0x7fee50032420 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:22.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 --2- 172.21.15.135:0/2994482668 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fee68060180 0x7fee680f6a00 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:22.690 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 >> 172.21.15.135:0/2994482668 conn(0x7fee680f09a0 msgr2=0x7fee680f1430 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:22.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 shutdown_connections 2023-11-15T06:44:22.691 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:22.686+0000 7fee67fff700 1 -- 172.21.15.135:0/2994482668 wait complete. 2023-11-15T06:44:22.691 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:23.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:22.293456+0000 mgr.smithi123.razjtn (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:23.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:22.685655+0000 mon.smithi123 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.135:0/2994482668' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:24.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:24.079 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:25.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.249+0000 7f8d42d10700 1 -- 172.21.15.135:0/2869199094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 msgr2=0x7f8d3c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:25.254 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.249+0000 7f8d42d10700 1 --2- 172.21.15.135:0/2869199094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c0f6930 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f8d30009fb0 tx=0x7f8d30009b30).stop 2023-11-15T06:44:25.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 -- 172.21.15.135:0/2869199094 shutdown_connections 2023-11-15T06:44:25.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 --2- 172.21.15.135:0/2869199094 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c0f6930 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:25.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 -- 172.21.15.135:0/2869199094 >> 172.21.15.135:0/2869199094 conn(0x7f8d3c0f1c70 msgr2=0x7f8d3c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:25.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 -- 172.21.15.135:0/2869199094 shutdown_connections 2023-11-15T06:44:25.255 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 -- 172.21.15.135:0/2869199094 wait complete. 2023-11-15T06:44:25.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 Processor -- start 2023-11-15T06:44:25.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 -- start start 2023-11-15T06:44:25.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:25.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.250+0000 7f8d42d10700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8d3c106c20 con 0x7f8d3c0f6510 2023-11-15T06:44:25.256 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d40aac700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:25.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d40aac700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33244/0 (socket says 172.21.15.135:33244) 2023-11-15T06:44:25.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d40aac700 1 -- 172.21.15.135:0/3953831238 learned_addr learned my addr 172.21.15.135:0/3953831238 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:25.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d40aac700 1 -- 172.21.15.135:0/3953831238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8d30005040 con 0x7f8d3c0f6510 2023-11-15T06:44:25.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d40aac700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f8d30014000 tx=0x7f8d3000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:25.257 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d397fa700 1 -- 172.21.15.135:0/3953831238 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d300095f0 con 0x7f8d3c0f6510 2023-11-15T06:44:25.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d397fa700 1 -- 172.21.15.135:0/3953831238 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8d30007e70 con 0x7f8d3c0f6510 2023-11-15T06:44:25.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8d3c106e20 con 0x7f8d3c0f6510 2023-11-15T06:44:25.258 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d397fa700 1 -- 172.21.15.135:0/3953831238 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8d30014ea0 con 0x7f8d3c0f6510 2023-11-15T06:44:25.259 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.251+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8d3c1072b0 con 0x7f8d3c0f6510 2023-11-15T06:44:25.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.252+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8d3c045ad0 con 0x7f8d3c0f6510 2023-11-15T06:44:25.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.252+0000 7f8d397fa700 1 -- 172.21.15.135:0/3953831238 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8d30025070 con 0x7f8d3c0f6510 2023-11-15T06:44:25.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.252+0000 7f8d397fa700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8d2c02ffb0 0x7f8d2c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:25.260 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.252+0000 7f8d397fa700 1 -- 172.21.15.135:0/3953831238 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8d30040930 con 0x7f8d3c0f6510 2023-11-15T06:44:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.253+0000 7f8d3bfff700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8d2c02ffb0 0x7f8d2c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.253+0000 7f8d3bfff700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8d2c02ffb0 0x7f8d2c032470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8d280096a0 tx=0x7f8d28006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:25.261 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.257+0000 7f8d397fa700 1 -- 172.21.15.135:0/3953831238 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8d300079c0 con 0x7f8d3c0f6510 2023-11-15T06:44:25.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.465+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8d3c0fbf60 con 0x7f8d3c0f6510 2023-11-15T06:44:25.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.465+0000 7f8d397fa700 1 -- 172.21.15.135:0/3953831238 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8d3000f300 con 0x7f8d3c0f6510 2023-11-15T06:44:25.469 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:25.469 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:25.470 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8d2c02ffb0 msgr2=0x7f8d2c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:25.471 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8d2c02ffb0 0x7f8d2c032470 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8d280096a0 tx=0x7f8d28006b40).stop 2023-11-15T06:44:25.471 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 msgr2=0x7f8d3c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:25.471 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f8d30014000 tx=0x7f8d3000b040).stop 2023-11-15T06:44:25.471 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 shutdown_connections 2023-11-15T06:44:25.471 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8d2c02ffb0 0x7f8d2c032470 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:25.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 --2- 172.21.15.135:0/3953831238 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8d3c0f6510 0x7f8d3c106730 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:25.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 >> 172.21.15.135:0/3953831238 conn(0x7f8d3c0f1c70 msgr2=0x7f8d3c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:25.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 shutdown_connections 2023-11-15T06:44:25.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:25.467+0000 7f8d42d10700 1 -- 172.21.15.135:0/3953831238 wait complete. 2023-11-15T06:44:25.472 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:24.293804+0000 mgr.smithi123.razjtn (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:25.466638+0000 mon.smithi123 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.135:0/3953831238' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:26.854 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:26.854 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:27.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:26.294134+0000 mgr.smithi123.razjtn (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:28.144 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.141+0000 7fbba5ab4700 1 -- 172.21.15.135:0/2310595501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 msgr2=0x7fbba00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:28.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.141+0000 7fbba5ab4700 1 --2- 172.21.15.135:0/2310595501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba00f9150 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fbb90009fb0 tx=0x7fbb90009b30).stop 2023-11-15T06:44:28.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.141+0000 7fbba5ab4700 1 -- 172.21.15.135:0/2310595501 shutdown_connections 2023-11-15T06:44:28.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.141+0000 7fbba5ab4700 1 --2- 172.21.15.135:0/2310595501 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba00f9150 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:28.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.141+0000 7fbba5ab4700 1 -- 172.21.15.135:0/2310595501 >> 172.21.15.135:0/2310595501 conn(0x7fbba00f0b70 msgr2=0x7fbba00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:28.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.141+0000 7fbba5ab4700 1 -- 172.21.15.135:0/2310595501 shutdown_connections 2023-11-15T06:44:28.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.141+0000 7fbba5ab4700 1 -- 172.21.15.135:0/2310595501 wait complete. 2023-11-15T06:44:28.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.142+0000 7fbba5ab4700 1 Processor -- start 2023-11-15T06:44:28.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.142+0000 7fbba5ab4700 1 -- start start 2023-11-15T06:44:28.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.142+0000 7fbba5ab4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:28.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.142+0000 7fbba5ab4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbba0106bd0 con 0x7fbba0060220 2023-11-15T06:44:28.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.142+0000 7fbb9f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:28.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.142+0000 7fbb9f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33258/0 (socket says 172.21.15.135:33258) 2023-11-15T06:44:28.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.142+0000 7fbb9f7fe700 1 -- 172.21.15.135:0/1960306112 learned_addr learned my addr 172.21.15.135:0/1960306112 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:28.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.143+0000 7fbb9f7fe700 1 -- 172.21.15.135:0/1960306112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbb90005040 con 0x7fbba0060220 2023-11-15T06:44:28.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.143+0000 7fbb9f7fe700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba01066e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbb90013510 tx=0x7fbb9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:28.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.143+0000 7fbb97fff700 1 -- 172.21.15.135:0/1960306112 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb90009320 con 0x7fbba0060220 2023-11-15T06:44:28.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.143+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbba0106dd0 con 0x7fbba0060220 2023-11-15T06:44:28.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.143+0000 7fbb97fff700 1 -- 172.21.15.135:0/1960306112 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbb90013f00 con 0x7fbba0060220 2023-11-15T06:44:28.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.143+0000 7fbb97fff700 1 -- 172.21.15.135:0/1960306112 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbb90014dc0 con 0x7fbba0060220 2023-11-15T06:44:28.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.143+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbba0107260 con 0x7fbba0060220 2023-11-15T06:44:28.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.144+0000 7fbb97fff700 1 -- 172.21.15.135:0/1960306112 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbb90014420 con 0x7fbba0060220 2023-11-15T06:44:28.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.144+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbba0045ad0 con 0x7fbba0060220 2023-11-15T06:44:28.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.144+0000 7fbb97fff700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb8c02ff60 0x7fbb8c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:28.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.144+0000 7fbb97fff700 1 -- 172.21.15.135:0/1960306112 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbb90041050 con 0x7fbba0060220 2023-11-15T06:44:28.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.144+0000 7fbb9effd700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb8c02ff60 0x7fbb8c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:28.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.145+0000 7fbb9effd700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb8c02ff60 0x7fbb8c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fbb880096a0 tx=0x7fbb88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:28.152 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.149+0000 7fbb97fff700 1 -- 172.21.15.135:0/1960306112 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbb900095f0 con 0x7fbba0060220 2023-11-15T06:44:28.359 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.356+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbba0108110 con 0x7fbba0060220 2023-11-15T06:44:28.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.357+0000 7fbb97fff700 1 -- 172.21.15.135:0/1960306112 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbb9001cb10 con 0x7fbba0060220 2023-11-15T06:44:28.360 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:28.360 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:28.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.358+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb8c02ff60 msgr2=0x7fbb8c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:28.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.358+0000 7fbba5ab4700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb8c02ff60 0x7fbb8c032420 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fbb880096a0 tx=0x7fbb88006b40).stop 2023-11-15T06:44:28.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 msgr2=0x7fbba01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:28.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba01066e0 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fbb90013510 tx=0x7fbb9000b040).stop 2023-11-15T06:44:28.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 shutdown_connections 2023-11-15T06:44:28.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fbb8c02ff60 0x7fbb8c032420 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:28.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 --2- 172.21.15.135:0/1960306112 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fbba0060220 0x7fbba01066e0 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:28.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 >> 172.21.15.135:0/1960306112 conn(0x7fbba00f0b70 msgr2=0x7fbba00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:28.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 shutdown_connections 2023-11-15T06:44:28.363 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:28.359+0000 7fbba5ab4700 1 -- 172.21.15.135:0/1960306112 wait complete. 2023-11-15T06:44:28.364 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:28.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:28 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:28.357891+0000 mon.smithi123 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.135:0/1960306112' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:29.741 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:29.741 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:29.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:28.294381+0000 mgr.smithi123.razjtn (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:30.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.896+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/2631889734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 msgr2=0x7f1e140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:30.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.896+0000 7f1e1c3d5700 1 --2- 172.21.15.135:0/2631889734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140f5830 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7f1e04009fb0 tx=0x7f1e04009b30).stop 2023-11-15T06:44:30.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.896+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/2631889734 shutdown_connections 2023-11-15T06:44:30.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.896+0000 7f1e1c3d5700 1 --2- 172.21.15.135:0/2631889734 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140f5830 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:30.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.896+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/2631889734 >> 172.21.15.135:0/2631889734 conn(0x7f1e140f0b70 msgr2=0x7f1e140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:30.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.896+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/2631889734 shutdown_connections 2023-11-15T06:44:30.902 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.896+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/2631889734 wait complete. 2023-11-15T06:44:30.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1c3d5700 1 Processor -- start 2023-11-15T06:44:30.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1c3d5700 1 -- start start 2023-11-15T06:44:30.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1c3d5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:30.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1c3d5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1e140fb820 con 0x7f1e140f5410 2023-11-15T06:44:30.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1a171700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:30.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1a171700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48618/0 (socket says 172.21.15.135:48618) 2023-11-15T06:44:30.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1a171700 1 -- 172.21.15.135:0/1900130208 learned_addr learned my addr 172.21.15.135:0/1900130208 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:30.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.897+0000 7f1e1a171700 1 -- 172.21.15.135:0/1900130208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1e04005040 con 0x7f1e140f5410 2023-11-15T06:44:30.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.898+0000 7f1e1a171700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140fd3a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f1e04013510 tx=0x7f1e0400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:30.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.898+0000 7f1e0affd700 1 -- 172.21.15.135:0/1900130208 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e04009320 con 0x7f1e140f5410 2023-11-15T06:44:30.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.898+0000 7f1e0affd700 1 -- 172.21.15.135:0/1900130208 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1e04013f00 con 0x7f1e140f5410 2023-11-15T06:44:30.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.898+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1e140fb9c0 con 0x7f1e140f5410 2023-11-15T06:44:30.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.898+0000 7f1e0affd700 1 -- 172.21.15.135:0/1900130208 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1e04014d90 con 0x7f1e140f5410 2023-11-15T06:44:30.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.898+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1e140fbe50 con 0x7f1e140f5410 2023-11-15T06:44:30.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.899+0000 7f1e0affd700 1 -- 172.21.15.135:0/1900130208 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1e040095f0 con 0x7f1e140f5410 2023-11-15T06:44:30.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.899+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1e14059100 con 0x7f1e140f5410 2023-11-15T06:44:30.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.899+0000 7f1e0affd700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1e0002ff60 0x7f1e00032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:30.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.899+0000 7f1e0affd700 1 -- 172.21.15.135:0/1900130208 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1e04041000 con 0x7f1e140f5410 2023-11-15T06:44:30.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.899+0000 7f1e19970700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1e0002ff60 0x7f1e00032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:30.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.899+0000 7f1e19970700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1e0002ff60 0x7f1e00032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1e100096a0 tx=0x7f1e10006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:30.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:30.903+0000 7f1e0affd700 1 -- 172.21.15.135:0/1900130208 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1e04014420 con 0x7f1e140f5410 2023-11-15T06:44:31.113 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.111+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1e140fae60 con 0x7f1e140f5410 2023-11-15T06:44:31.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.111+0000 7f1e0affd700 1 -- 172.21.15.135:0/1900130208 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1e0401d7f0 con 0x7f1e140f5410 2023-11-15T06:44:31.115 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:31.115 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:31.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1e0002ff60 msgr2=0x7f1e00032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:31.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1e0002ff60 0x7f1e00032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7f1e100096a0 tx=0x7f1e10006b40).stop 2023-11-15T06:44:31.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 msgr2=0x7f1e140fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:31.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140fd3a0 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7f1e04013510 tx=0x7f1e0400b040).stop 2023-11-15T06:44:31.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 shutdown_connections 2023-11-15T06:44:31.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f1e0002ff60 0x7f1e00032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:31.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 --2- 172.21.15.135:0/1900130208 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f1e140f5410 0x7f1e140fd3a0 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:31.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 >> 172.21.15.135:0/1900130208 conn(0x7f1e140f0b70 msgr2=0x7f1e1405fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:31.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 shutdown_connections 2023-11-15T06:44:31.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:31.113+0000 7f1e1c3d5700 1 -- 172.21.15.135:0/1900130208 wait complete. 2023-11-15T06:44:31.118 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:31.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:30.294628+0000 mgr.smithi123.razjtn (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:31.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:31.112454+0000 mon.smithi123 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.135:0/1900130208' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:32.490 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:32.491 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:33.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.632+0000 7f53f810d700 1 -- 172.21.15.135:0/2433054119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 msgr2=0x7f53f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:33.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.632+0000 7f53f810d700 1 --2- 172.21.15.135:0/2433054119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f00f5830 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7f53e0009fb0 tx=0x7f53e0009b30).stop 2023-11-15T06:44:33.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.632+0000 7f53f810d700 1 -- 172.21.15.135:0/2433054119 shutdown_connections 2023-11-15T06:44:33.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.632+0000 7f53f810d700 1 --2- 172.21.15.135:0/2433054119 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f00f5830 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:33.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.632+0000 7f53f810d700 1 -- 172.21.15.135:0/2433054119 >> 172.21.15.135:0/2433054119 conn(0x7f53f00f0b70 msgr2=0x7f53f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:33.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.632+0000 7f53f810d700 1 -- 172.21.15.135:0/2433054119 shutdown_connections 2023-11-15T06:44:33.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.632+0000 7f53f810d700 1 -- 172.21.15.135:0/2433054119 wait complete. 2023-11-15T06:44:33.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f810d700 1 Processor -- start 2023-11-15T06:44:33.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f810d700 1 -- start start 2023-11-15T06:44:33.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f810d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:33.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f810d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53f0060220 con 0x7f53f00f5410 2023-11-15T06:44:33.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f5ea9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:33.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f5ea9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48648/0 (socket says 172.21.15.135:48648) 2023-11-15T06:44:33.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f5ea9700 1 -- 172.21.15.135:0/4137069713 learned_addr learned my addr 172.21.15.135:0/4137069713 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:33.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f5ea9700 1 -- 172.21.15.135:0/4137069713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f53e0005040 con 0x7f53f00f5410 2023-11-15T06:44:33.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.633+0000 7f53f5ea9700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f01093f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f53e0013510 tx=0x7f53e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:33.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.634+0000 7f53e67fc700 1 -- 172.21.15.135:0/4137069713 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e0009320 con 0x7f53f00f5410 2023-11-15T06:44:33.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.634+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53f0060420 con 0x7f53f00f5410 2023-11-15T06:44:33.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.634+0000 7f53e67fc700 1 -- 172.21.15.135:0/4137069713 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f53e0013f00 con 0x7f53f00f5410 2023-11-15T06:44:33.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.634+0000 7f53e67fc700 1 -- 172.21.15.135:0/4137069713 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f53e0014d90 con 0x7f53f00f5410 2023-11-15T06:44:33.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.634+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53f0109aa0 con 0x7f53f00f5410 2023-11-15T06:44:33.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.635+0000 7f53e67fc700 1 -- 172.21.15.135:0/4137069713 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f53e00095f0 con 0x7f53f00f5410 2023-11-15T06:44:33.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.635+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53f0059100 con 0x7f53f00f5410 2023-11-15T06:44:33.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.635+0000 7f53e67fc700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f53dc02ffb0 0x7f53dc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:33.644 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.635+0000 7f53e67fc700 1 -- 172.21.15.135:0/4137069713 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f53e0041080 con 0x7f53f00f5410 2023-11-15T06:44:33.644 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.635+0000 7f53f56a8700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f53dc02ffb0 0x7f53dc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:33.644 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.635+0000 7f53f56a8700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f53dc02ffb0 0x7f53dc032470 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f53ec0096a0 tx=0x7f53ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:33.645 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.641+0000 7f53e67fc700 1 -- 172.21.15.135:0/4137069713 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f53e006d2b0 con 0x7f53f00f5410 2023-11-15T06:44:33.849 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.846+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53f00fae60 con 0x7f53f00f5410 2023-11-15T06:44:33.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.847+0000 7f53e67fc700 1 -- 172.21.15.135:0/4137069713 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f53e0014420 con 0x7f53f00f5410 2023-11-15T06:44:33.850 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:33.851 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:33.856 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f53dc02ffb0 msgr2=0x7f53dc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:33.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f53dc02ffb0 0x7f53dc032470 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f53ec0096a0 tx=0x7f53ec006b40).stop 2023-11-15T06:44:33.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 msgr2=0x7f53f01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:33.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f01093f0 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7f53e0013510 tx=0x7f53e000b040).stop 2023-11-15T06:44:33.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 shutdown_connections 2023-11-15T06:44:33.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f53dc02ffb0 0x7f53dc032470 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:33.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 --2- 172.21.15.135:0/4137069713 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f53f00f5410 0x7f53f01093f0 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:33.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 >> 172.21.15.135:0/4137069713 conn(0x7f53f00f0b70 msgr2=0x7f53f00f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:33.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 shutdown_connections 2023-11-15T06:44:33.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:33.849+0000 7f53f810d700 1 -- 172.21.15.135:0/4137069713 wait complete. 2023-11-15T06:44:33.859 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:33.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:32.294983+0000 mgr.smithi123.razjtn (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:34.866 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:34 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:33.848351+0000 mon.smithi123 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.135:0/4137069713' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:35.233 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:35.233 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:35.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:34.295299+0000 mgr.smithi123.razjtn (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:36.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 -- 172.21.15.135:0/621760279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 msgr2=0x7f91100f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:36.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 --2- 172.21.15.135:0/621760279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91100f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f9104009fb0 tx=0x7f9104009b30).stop 2023-11-15T06:44:36.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 -- 172.21.15.135:0/621760279 shutdown_connections 2023-11-15T06:44:36.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 --2- 172.21.15.135:0/621760279 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91100f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:36.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 -- 172.21.15.135:0/621760279 >> 172.21.15.135:0/621760279 conn(0x7f91100f09a0 msgr2=0x7f91100f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:36.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 -- 172.21.15.135:0/621760279 shutdown_connections 2023-11-15T06:44:36.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 -- 172.21.15.135:0/621760279 wait complete. 2023-11-15T06:44:36.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 Processor -- start 2023-11-15T06:44:36.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.383+0000 7f9114f60700 1 -- start start 2023-11-15T06:44:36.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f9114f60700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91101064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:36.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f9114f60700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f91101069b0 con 0x7f91100f51a0 2023-11-15T06:44:36.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f910f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91101064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:36.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f910f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91101064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:48660/0 (socket says 172.21.15.135:48660) 2023-11-15T06:44:36.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f910f7fe700 1 -- 172.21.15.135:0/3968071320 learned_addr learned my addr 172.21.15.135:0/3968071320 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:36.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f910f7fe700 1 -- 172.21.15.135:0/3968071320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9104005040 con 0x7f91100f51a0 2023-11-15T06:44:36.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f910f7fe700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91101064c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9104013510 tx=0x7f910400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:36.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f90f7fff700 1 -- 172.21.15.135:0/3968071320 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9104009320 con 0x7f91100f51a0 2023-11-15T06:44:36.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9110106bb0 con 0x7f91100f51a0 2023-11-15T06:44:36.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f90f7fff700 1 -- 172.21.15.135:0/3968071320 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9104013f00 con 0x7f91100f51a0 2023-11-15T06:44:36.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f90f7fff700 1 -- 172.21.15.135:0/3968071320 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9104014dc0 con 0x7f91100f51a0 2023-11-15T06:44:36.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.384+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9110107040 con 0x7f91100f51a0 2023-11-15T06:44:36.392 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.385+0000 7f90f7fff700 1 -- 172.21.15.135:0/3968071320 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f91040095f0 con 0x7f91100f51a0 2023-11-15T06:44:36.392 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.385+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9110045ad0 con 0x7f91100f51a0 2023-11-15T06:44:36.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.386+0000 7f90f7fff700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f90f802ff60 0x7f90f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:36.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.386+0000 7f90f7fff700 1 -- 172.21.15.135:0/3968071320 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9104041000 con 0x7f91100f51a0 2023-11-15T06:44:36.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.386+0000 7f910effd700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f90f802ff60 0x7f90f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:36.393 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.386+0000 7f910effd700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f90f802ff60 0x7f90f8032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f91000096a0 tx=0x7f9100006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:36.394 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.390+0000 7f90f7fff700 1 -- 172.21.15.135:0/3968071320 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9104014420 con 0x7f91100f51a0 2023-11-15T06:44:36.599 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.597+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9110107ef0 con 0x7f91100f51a0 2023-11-15T06:44:36.600 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.597+0000 7f90f7fff700 1 -- 172.21.15.135:0/3968071320 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9104010230 con 0x7f91100f51a0 2023-11-15T06:44:36.600 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:36.601 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:36.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f90f802ff60 msgr2=0x7f90f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:36.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f90f802ff60 0x7f90f8032420 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f91000096a0 tx=0x7f9100006b40).stop 2023-11-15T06:44:36.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 msgr2=0x7f91101064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:36.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91101064c0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f9104013510 tx=0x7f910400b040).stop 2023-11-15T06:44:36.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 shutdown_connections 2023-11-15T06:44:36.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f90f802ff60 0x7f90f8032420 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:36.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 --2- 172.21.15.135:0/3968071320 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f91100f51a0 0x7f91101064c0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:36.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 >> 172.21.15.135:0/3968071320 conn(0x7f91100f09a0 msgr2=0x7f91100f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:36.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 shutdown_connections 2023-11-15T06:44:36.604 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:36.599+0000 7f9114f60700 1 -- 172.21.15.135:0/3968071320 wait complete. 2023-11-15T06:44:36.604 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:37.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:36.295609+0000 mgr.smithi123.razjtn (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:37.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:36.598400+0000 mon.smithi123 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.135:0/3968071320' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:37.976 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:37.976 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:39.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.094+0000 7f4536acc700 1 -- 172.21.15.135:0/206497463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 msgr2=0x7f45300f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:39.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.094+0000 7f4536acc700 1 --2- 172.21.15.135:0/206497463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f45300f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f4524009fb0 tx=0x7f4524009b30).stop 2023-11-15T06:44:39.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.094+0000 7f4536acc700 1 -- 172.21.15.135:0/206497463 shutdown_connections 2023-11-15T06:44:39.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.094+0000 7f4536acc700 1 --2- 172.21.15.135:0/206497463 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f45300f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:39.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.094+0000 7f4536acc700 1 -- 172.21.15.135:0/206497463 >> 172.21.15.135:0/206497463 conn(0x7f45300f1c70 msgr2=0x7f45300f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:39.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.094+0000 7f4536acc700 1 -- 172.21.15.135:0/206497463 shutdown_connections 2023-11-15T06:44:39.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.094+0000 7f4536acc700 1 -- 172.21.15.135:0/206497463 wait complete. 2023-11-15T06:44:39.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.095+0000 7f4536acc700 1 Processor -- start 2023-11-15T06:44:39.101 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.095+0000 7f4536acc700 1 -- start start 2023-11-15T06:44:39.101 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.096+0000 7f4536acc700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f4530106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:39.102 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.096+0000 7f4536acc700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4530106c20 con 0x7f45300f6510 2023-11-15T06:44:39.104 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.096+0000 7f4534868700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f4530106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:39.104 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.096+0000 7f4534868700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f4530106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53558/0 (socket says 172.21.15.135:53558) 2023-11-15T06:44:39.104 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.096+0000 7f4534868700 1 -- 172.21.15.135:0/514757776 learned_addr learned my addr 172.21.15.135:0/514757776 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:39.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.096+0000 7f4534868700 1 -- 172.21.15.135:0/514757776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4524005040 con 0x7f45300f6510 2023-11-15T06:44:39.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.096+0000 7f4534868700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f4530106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f4524013510 tx=0x7f452400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:39.106 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.097+0000 7f452d7fa700 1 -- 172.21.15.135:0/514757776 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4524009320 con 0x7f45300f6510 2023-11-15T06:44:39.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.097+0000 7f452d7fa700 1 -- 172.21.15.135:0/514757776 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4524013f00 con 0x7f45300f6510 2023-11-15T06:44:39.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.097+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4530106e20 con 0x7f45300f6510 2023-11-15T06:44:39.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.097+0000 7f452d7fa700 1 -- 172.21.15.135:0/514757776 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4524014d90 con 0x7f45300f6510 2023-11-15T06:44:39.107 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.097+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f45301072b0 con 0x7f45300f6510 2023-11-15T06:44:39.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.097+0000 7f452d7fa700 1 -- 172.21.15.135:0/514757776 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f45240095f0 con 0x7f45300f6510 2023-11-15T06:44:39.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.097+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4530045ad0 con 0x7f45300f6510 2023-11-15T06:44:39.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.098+0000 7f452d7fa700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f452002ff60 0x7f4520032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:39.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.098+0000 7f452d7fa700 1 -- 172.21.15.135:0/514757776 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4524041000 con 0x7f45300f6510 2023-11-15T06:44:39.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.098+0000 7f452ffff700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f452002ff60 0x7f4520032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:39.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.098+0000 7f452ffff700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f452002ff60 0x7f4520032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f451c0096a0 tx=0x7f451c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:39.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.102+0000 7f452d7fa700 1 -- 172.21.15.135:0/514757776 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f452406d2b0 con 0x7f45300f6510 2023-11-15T06:44:39.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.306+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f45300fc050 con 0x7f45300f6510 2023-11-15T06:44:39.310 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.307+0000 7f452d7fa700 1 -- 172.21.15.135:0/514757776 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f452401c320 con 0x7f45300f6510 2023-11-15T06:44:39.310 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:39.311 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:39.312 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f452002ff60 msgr2=0x7f4520032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:39.312 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f452002ff60 0x7f4520032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f451c0096a0 tx=0x7f451c006b40).stop 2023-11-15T06:44:39.312 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 msgr2=0x7f4530106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:39.312 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f4530106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f4524013510 tx=0x7f452400b040).stop 2023-11-15T06:44:39.312 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 shutdown_connections 2023-11-15T06:44:39.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f452002ff60 0x7f4520032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:39.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 --2- 172.21.15.135:0/514757776 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f45300f6510 0x7f4530106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:39.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 >> 172.21.15.135:0/514757776 conn(0x7f45300f1c70 msgr2=0x7f45300f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:39.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 shutdown_connections 2023-11-15T06:44:39.313 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:39.308+0000 7f4536acc700 1 -- 172.21.15.135:0/514757776 wait complete. 2023-11-15T06:44:39.314 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:39.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:38.295914+0000 mgr.smithi123.razjtn (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:39.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:39.307990+0000 mon.smithi123 (mon.0) 279 : audit [DBG] from='client.? 172.21.15.135:0/514757776' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:40.707 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:40.708 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:40.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:40.369838+0000 mon.smithi123 (mon.0) 280 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:44:40.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:40.375285+0000 mon.smithi123 (mon.0) 281 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:44:41.840 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.836+0000 7f96136af700 1 -- 172.21.15.135:0/4139356594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 msgr2=0x7f960c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:41.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.836+0000 7f96136af700 1 --2- 172.21.15.135:0/4139356594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c0f5830 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f95fc009fb0 tx=0x7f95fc009b30).stop 2023-11-15T06:44:41.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.836+0000 7f96136af700 1 -- 172.21.15.135:0/4139356594 shutdown_connections 2023-11-15T06:44:41.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.836+0000 7f96136af700 1 --2- 172.21.15.135:0/4139356594 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c0f5830 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:41.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.836+0000 7f96136af700 1 -- 172.21.15.135:0/4139356594 >> 172.21.15.135:0/4139356594 conn(0x7f960c0f0b70 msgr2=0x7f960c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:41.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.836+0000 7f96136af700 1 -- 172.21.15.135:0/4139356594 shutdown_connections 2023-11-15T06:44:41.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.836+0000 7f96136af700 1 -- 172.21.15.135:0/4139356594 wait complete. 2023-11-15T06:44:41.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f96136af700 1 Processor -- start 2023-11-15T06:44:41.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f96136af700 1 -- start start 2023-11-15T06:44:41.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f96136af700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:41.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f96136af700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f960c106c20 con 0x7f960c0f5410 2023-11-15T06:44:41.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f961144b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:41.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f961144b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53586/0 (socket says 172.21.15.135:53586) 2023-11-15T06:44:41.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f961144b700 1 -- 172.21.15.135:0/3340250853 learned_addr learned my addr 172.21.15.135:0/3340250853 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:41.843 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.837+0000 7f961144b700 1 -- 172.21.15.135:0/3340250853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f95fc005040 con 0x7f960c0f5410 2023-11-15T06:44:41.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.838+0000 7f961144b700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c106730 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f95fc013510 tx=0x7f95fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:41.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.838+0000 7f9601ffb700 1 -- 172.21.15.135:0/3340250853 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95fc009320 con 0x7f960c0f5410 2023-11-15T06:44:41.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.838+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f960c106e20 con 0x7f960c0f5410 2023-11-15T06:44:41.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.838+0000 7f9601ffb700 1 -- 172.21.15.135:0/3340250853 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f95fc013f00 con 0x7f960c0f5410 2023-11-15T06:44:41.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.838+0000 7f9601ffb700 1 -- 172.21.15.135:0/3340250853 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f95fc014d90 con 0x7f960c0f5410 2023-11-15T06:44:41.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.838+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f960c1072b0 con 0x7f960c0f5410 2023-11-15T06:44:41.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.839+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f960c045ad0 con 0x7f960c0f5410 2023-11-15T06:44:41.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.839+0000 7f9601ffb700 1 -- 172.21.15.135:0/3340250853 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f95fc0095f0 con 0x7f960c0f5410 2023-11-15T06:44:41.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.839+0000 7f9601ffb700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95f802ffb0 0x7f95f8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:41.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.839+0000 7f9601ffb700 1 -- 172.21.15.135:0/3340250853 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f95fc041080 con 0x7f960c0f5410 2023-11-15T06:44:41.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.839+0000 7f9610c4a700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95f802ffb0 0x7f95f8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:41.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.840+0000 7f9610c4a700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95f802ffb0 0x7f95f8032470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:41.847 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:41.844+0000 7f9601ffb700 1 -- 172.21.15.135:0/3340250853 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f95fc06d2b0 con 0x7f960c0f5410 2023-11-15T06:44:41.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:40.296146+0000 mgr.smithi123.razjtn (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:42.052 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.049+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f960c0fae60 con 0x7f960c0f5410 2023-11-15T06:44:42.053 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.050+0000 7f9601ffb700 1 -- 172.21.15.135:0/3340250853 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f95fc014420 con 0x7f960c0f5410 2023-11-15T06:44:42.053 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:42.054 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:42.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95f802ffb0 msgr2=0x7f95f8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:42.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95f802ffb0 0x7f95f8032470 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f96080096a0 tx=0x7f9608006b40).stop 2023-11-15T06:44:42.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 msgr2=0x7f960c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:42.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c106730 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f95fc013510 tx=0x7f95fc00b040).stop 2023-11-15T06:44:42.055 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 shutdown_connections 2023-11-15T06:44:42.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f95f802ffb0 0x7f95f8032470 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:42.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 --2- 172.21.15.135:0/3340250853 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f960c0f5410 0x7f960c106730 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:42.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 >> 172.21.15.135:0/3340250853 conn(0x7f960c0f0b70 msgr2=0x7f960c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:42.056 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 shutdown_connections 2023-11-15T06:44:42.057 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:42.052+0000 7f96136af700 1 -- 172.21.15.135:0/3340250853 wait complete. 2023-11-15T06:44:42.057 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:42.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:42 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:42.051085+0000 mon.smithi123 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.135:0/3340250853' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:43.438 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:43.438 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:43.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:42.296530+0000 mgr.smithi123.razjtn (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:44.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.582+0000 7faba1057700 1 -- 172.21.15.135:0/1399573184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 msgr2=0x7fab9c0f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:44.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.582+0000 7faba1057700 1 --2- 172.21.15.135:0/1399573184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c0f4be0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7fab90009fb0 tx=0x7fab90009b30).stop 2023-11-15T06:44:44.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.582+0000 7faba1057700 1 -- 172.21.15.135:0/1399573184 shutdown_connections 2023-11-15T06:44:44.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.582+0000 7faba1057700 1 --2- 172.21.15.135:0/1399573184 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c0f4be0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:44.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.582+0000 7faba1057700 1 -- 172.21.15.135:0/1399573184 >> 172.21.15.135:0/1399573184 conn(0x7fab9c0f02e0 msgr2=0x7fab9c0f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:44.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.582+0000 7faba1057700 1 -- 172.21.15.135:0/1399573184 shutdown_connections 2023-11-15T06:44:44.587 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.582+0000 7faba1057700 1 -- 172.21.15.135:0/1399573184 wait complete. 2023-11-15T06:44:44.587 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7faba1057700 1 Processor -- start 2023-11-15T06:44:44.587 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7faba1057700 1 -- start start 2023-11-15T06:44:44.589 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7faba1057700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:44.589 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7faba1057700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fab9c106bf0 con 0x7fab9c0f47c0 2023-11-15T06:44:44.589 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7fab9ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:44.589 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7fab9ad9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53606/0 (socket says 172.21.15.135:53606) 2023-11-15T06:44:44.589 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7fab9ad9d700 1 -- 172.21.15.135:0/2358402169 learned_addr learned my addr 172.21.15.135:0/2358402169 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:44.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.583+0000 7fab9ad9d700 1 -- 172.21.15.135:0/2358402169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fab90005040 con 0x7fab9c0f47c0 2023-11-15T06:44:44.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.584+0000 7fab9ad9d700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c106700 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fab90014000 tx=0x7fab9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:44.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.584+0000 7fab837fe700 1 -- 172.21.15.135:0/2358402169 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab900095f0 con 0x7fab9c0f47c0 2023-11-15T06:44:44.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.584+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fab9c106df0 con 0x7fab9c0f47c0 2023-11-15T06:44:44.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.584+0000 7fab837fe700 1 -- 172.21.15.135:0/2358402169 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fab90007e70 con 0x7fab9c0f47c0 2023-11-15T06:44:44.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.584+0000 7fab837fe700 1 -- 172.21.15.135:0/2358402169 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fab90014ea0 con 0x7fab9c0f47c0 2023-11-15T06:44:44.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.584+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fab9c107280 con 0x7fab9c0f47c0 2023-11-15T06:44:44.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.585+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fab9c045ad0 con 0x7fab9c0f47c0 2023-11-15T06:44:44.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.585+0000 7fab837fe700 1 -- 172.21.15.135:0/2358402169 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fab900079c0 con 0x7fab9c0f47c0 2023-11-15T06:44:44.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.585+0000 7fab837fe700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fab8402ff60 0x7fab84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:44.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.585+0000 7fab837fe700 1 -- 172.21.15.135:0/2358402169 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fab90040860 con 0x7fab9c0f47c0 2023-11-15T06:44:44.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.585+0000 7fab9a59c700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fab8402ff60 0x7fab84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:44.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.585+0000 7fab9a59c700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fab8402ff60 0x7fab84032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:44.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.589+0000 7fab837fe700 1 -- 172.21.15.135:0/2358402169 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fab9000f950 con 0x7fab9c0f47c0 2023-11-15T06:44:44.798 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.795+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fab9c108130 con 0x7fab9c0f47c0 2023-11-15T06:44:44.799 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.796+0000 7fab837fe700 1 -- 172.21.15.135:0/2358402169 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fab90010230 con 0x7fab9c0f47c0 2023-11-15T06:44:44.800 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:44.800 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:44.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fab8402ff60 msgr2=0x7fab84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:44.801 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fab8402ff60 0x7fab84032420 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7fab8c0096a0 tx=0x7fab8c006b40).stop 2023-11-15T06:44:44.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 msgr2=0x7fab9c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:44.802 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c106700 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7fab90014000 tx=0x7fab9000b040).stop 2023-11-15T06:44:44.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 shutdown_connections 2023-11-15T06:44:44.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fab8402ff60 0x7fab84032420 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:44.803 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 --2- 172.21.15.135:0/2358402169 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fab9c0f47c0 0x7fab9c106700 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:44.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 >> 172.21.15.135:0/2358402169 conn(0x7fab9c0f02e0 msgr2=0x7fab9c0f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:44.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 shutdown_connections 2023-11-15T06:44:44.804 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:44.797+0000 7faba1057700 1 -- 172.21.15.135:0/2358402169 wait complete. 2023-11-15T06:44:44.805 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:45.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:44.296925+0000 mgr.smithi123.razjtn (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:45.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:44.796974+0000 mon.smithi123 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.135:0/2358402169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:46.181 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:46.181 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:47.367 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 -- 172.21.15.135:0/618138869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 msgr2=0x7f8f780f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:47.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 --2- 172.21.15.135:0/618138869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f780f6930 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f8f6c009fb0 tx=0x7f8f6c009b30).stop 2023-11-15T06:44:47.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 -- 172.21.15.135:0/618138869 shutdown_connections 2023-11-15T06:44:47.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 --2- 172.21.15.135:0/618138869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f780f6930 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:47.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 -- 172.21.15.135:0/618138869 >> 172.21.15.135:0/618138869 conn(0x7f8f780f1c70 msgr2=0x7f8f780f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:47.368 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 -- 172.21.15.135:0/618138869 shutdown_connections 2023-11-15T06:44:47.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 -- 172.21.15.135:0/618138869 wait complete. 2023-11-15T06:44:47.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.364+0000 7f8f7ddef700 1 Processor -- start 2023-11-15T06:44:47.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.365+0000 7f8f7ddef700 1 -- start start 2023-11-15T06:44:47.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.365+0000 7f8f7ddef700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f7806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:47.369 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.365+0000 7f8f7ddef700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f78069e20 con 0x7f8f780f6510 2023-11-15T06:44:47.371 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.365+0000 7f8f777fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f7806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:47.371 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.365+0000 7f8f777fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f7806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53624/0 (socket says 172.21.15.135:53624) 2023-11-15T06:44:47.371 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.365+0000 7f8f777fe700 1 -- 172.21.15.135:0/1108988725 learned_addr learned my addr 172.21.15.135:0/1108988725 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:47.372 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.365+0000 7f8f777fe700 1 -- 172.21.15.135:0/1108988725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f6c005040 con 0x7f8f780f6510 2023-11-15T06:44:47.372 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.366+0000 7f8f777fe700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f7806b9a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f8f6c013510 tx=0x7f8f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:47.372 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.366+0000 7f8f5ffff700 1 -- 172.21.15.135:0/1108988725 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c009320 con 0x7f8f780f6510 2023-11-15T06:44:47.372 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.366+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f78069fc0 con 0x7f8f780f6510 2023-11-15T06:44:47.372 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.366+0000 7f8f5ffff700 1 -- 172.21.15.135:0/1108988725 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f6c013f00 con 0x7f8f780f6510 2023-11-15T06:44:47.373 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.366+0000 7f8f5ffff700 1 -- 172.21.15.135:0/1108988725 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f6c014d90 con 0x7f8f780f6510 2023-11-15T06:44:47.373 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.366+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f7806a450 con 0x7f8f780f6510 2023-11-15T06:44:47.373 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.367+0000 7f8f5ffff700 1 -- 172.21.15.135:0/1108988725 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f6c014420 con 0x7f8f780f6510 2023-11-15T06:44:47.373 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.367+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f78059100 con 0x7f8f780f6510 2023-11-15T06:44:47.373 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.367+0000 7f8f5ffff700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f6002ff60 0x7f8f60032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:47.374 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.367+0000 7f8f5ffff700 1 -- 172.21.15.135:0/1108988725 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f6c041110 con 0x7f8f780f6510 2023-11-15T06:44:47.374 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.367+0000 7f8f76ffd700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f6002ff60 0x7f8f60032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:47.374 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.367+0000 7f8f76ffd700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f6002ff60 0x7f8f60032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8f680096a0 tx=0x7f8f68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:47.375 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.371+0000 7f8f5ffff700 1 -- 172.21.15.135:0/1108988725 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f6c0095f0 con 0x7f8f780f6510 2023-11-15T06:44:47.581 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.579+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f78045ad0 con 0x7f8f780f6510 2023-11-15T06:44:47.582 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.579+0000 7f8f5ffff700 1 -- 172.21.15.135:0/1108988725 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f6c01db70 con 0x7f8f780f6510 2023-11-15T06:44:47.583 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:47.583 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:47.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f6002ff60 msgr2=0x7f8f60032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:47.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f6002ff60 0x7f8f60032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f8f680096a0 tx=0x7f8f68006b40).stop 2023-11-15T06:44:47.584 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 msgr2=0x7f8f7806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:47.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f7806b9a0 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f8f6c013510 tx=0x7f8f6c00b040).stop 2023-11-15T06:44:47.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 shutdown_connections 2023-11-15T06:44:47.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f8f6002ff60 0x7f8f60032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:47.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 --2- 172.21.15.135:0/1108988725 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f8f780f6510 0x7f8f7806b9a0 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:47.585 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 >> 172.21.15.135:0/1108988725 conn(0x7f8f780f1c70 msgr2=0x7f8f780f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:47.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 shutdown_connections 2023-11-15T06:44:47.586 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:47.581+0000 7f8f7ddef700 1 -- 172.21.15.135:0/1108988725 wait complete. 2023-11-15T06:44:47.586 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:47.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:46.297348+0000 mgr.smithi123.razjtn (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:47.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:47.580561+0000 mon.smithi123 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.135:0/1108988725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:48.960 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:48.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:48.297817+0000 mgr.smithi123.razjtn (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:50.092 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.088+0000 7fed7d152700 1 -- 172.21.15.135:0/4163452126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 msgr2=0x7fed780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:50.092 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.088+0000 7fed7d152700 1 --2- 172.21.15.135:0/4163452126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed780f5830 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7fed60009fb0 tx=0x7fed60009b30).stop 2023-11-15T06:44:50.093 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.088+0000 7fed7d152700 1 -- 172.21.15.135:0/4163452126 shutdown_connections 2023-11-15T06:44:50.093 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.088+0000 7fed7d152700 1 --2- 172.21.15.135:0/4163452126 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed780f5830 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:50.093 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.088+0000 7fed7d152700 1 -- 172.21.15.135:0/4163452126 >> 172.21.15.135:0/4163452126 conn(0x7fed780f0b70 msgr2=0x7fed780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:50.093 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.088+0000 7fed7d152700 1 -- 172.21.15.135:0/4163452126 shutdown_connections 2023-11-15T06:44:50.093 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.088+0000 7fed7d152700 1 -- 172.21.15.135:0/4163452126 wait complete. 2023-11-15T06:44:50.094 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.089+0000 7fed7d152700 1 Processor -- start 2023-11-15T06:44:50.094 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.089+0000 7fed7d152700 1 -- start start 2023-11-15T06:44:50.094 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.089+0000 7fed7d152700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed78109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:50.094 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.089+0000 7fed7d152700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed78060240 con 0x7fed780f5410 2023-11-15T06:44:50.095 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.089+0000 7fed76d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed78109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:50.095 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.089+0000 7fed76d9d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed78109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:50644/0 (socket says 172.21.15.135:50644) 2023-11-15T06:44:50.095 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.089+0000 7fed76d9d700 1 -- 172.21.15.135:0/3861535386 learned_addr learned my addr 172.21.15.135:0/3861535386 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:50.097 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.090+0000 7fed76d9d700 1 -- 172.21.15.135:0/3861535386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed60005040 con 0x7fed780f5410 2023-11-15T06:44:50.097 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.090+0000 7fed76d9d700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed78109430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fed60014000 tx=0x7fed6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:50.097 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.090+0000 7fed6f7fe700 1 -- 172.21.15.135:0/3861535386 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed600095f0 con 0x7fed780f5410 2023-11-15T06:44:50.097 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.090+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed78060440 con 0x7fed780f5410 2023-11-15T06:44:50.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.090+0000 7fed6f7fe700 1 -- 172.21.15.135:0/3861535386 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed60007e70 con 0x7fed780f5410 2023-11-15T06:44:50.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.090+0000 7fed6f7fe700 1 -- 172.21.15.135:0/3861535386 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed60014ea0 con 0x7fed780f5410 2023-11-15T06:44:50.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.090+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed78109ae0 con 0x7fed780f5410 2023-11-15T06:44:50.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.091+0000 7fed6f7fe700 1 -- 172.21.15.135:0/3861535386 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed60025070 con 0x7fed780f5410 2023-11-15T06:44:50.098 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.091+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed78045ad0 con 0x7fed780f5410 2023-11-15T06:44:50.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.091+0000 7fed6f7fe700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6402ff60 0x7fed64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:50.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.091+0000 7fed6f7fe700 1 -- 172.21.15.135:0/3861535386 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed600409f0 con 0x7fed780f5410 2023-11-15T06:44:50.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.091+0000 7fed7659c700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6402ff60 0x7fed64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:50.099 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.092+0000 7fed7659c700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6402ff60 0x7fed64032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fed680096a0 tx=0x7fed68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:50.100 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.096+0000 7fed6f7fe700 1 -- 172.21.15.135:0/3861535386 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed6000f4e0 con 0x7fed780f5410 2023-11-15T06:44:50.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.301+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed780fae60 con 0x7fed780f5410 2023-11-15T06:44:50.305 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.302+0000 7fed6f7fe700 1 -- 172.21.15.135:0/3861535386 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed6000f300 con 0x7fed780f5410 2023-11-15T06:44:50.306 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:50.306 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:50.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6402ff60 msgr2=0x7fed64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:50.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6402ff60 0x7fed64032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7fed680096a0 tx=0x7fed68006b40).stop 2023-11-15T06:44:50.307 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 msgr2=0x7fed78109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:50.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed78109430 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7fed60014000 tx=0x7fed6000b040).stop 2023-11-15T06:44:50.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 shutdown_connections 2023-11-15T06:44:50.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fed6402ff60 0x7fed64032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:50.308 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 --2- 172.21.15.135:0/3861535386 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fed780f5410 0x7fed78109430 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:50.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 >> 172.21.15.135:0/3861535386 conn(0x7fed780f0b70 msgr2=0x7fed780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:50.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 shutdown_connections 2023-11-15T06:44:50.309 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:50.304+0000 7fed7d152700 1 -- 172.21.15.135:0/3861535386 wait complete. 2023-11-15T06:44:50.309 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:50.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:50 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:50.303330+0000 mon.smithi123 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.135:0/3861535386' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:51.686 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:51.686 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:51.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:50.298215+0000 mgr.smithi123.razjtn (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:52.807 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.804+0000 7f3050644700 1 -- 172.21.15.135:0/2204841805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 msgr2=0x7f30480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:52.808 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.804+0000 7f3050644700 1 --2- 172.21.15.135:0/2204841805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f30480f5830 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f3038009fb0 tx=0x7f3038009b30).stop 2023-11-15T06:44:52.808 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.805+0000 7f3050644700 1 -- 172.21.15.135:0/2204841805 shutdown_connections 2023-11-15T06:44:52.808 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.805+0000 7f3050644700 1 --2- 172.21.15.135:0/2204841805 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f30480f5830 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:52.809 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.805+0000 7f3050644700 1 -- 172.21.15.135:0/2204841805 >> 172.21.15.135:0/2204841805 conn(0x7f30480f0b70 msgr2=0x7f30480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:52.809 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.805+0000 7f3050644700 1 -- 172.21.15.135:0/2204841805 shutdown_connections 2023-11-15T06:44:52.809 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.805+0000 7f3050644700 1 -- 172.21.15.135:0/2204841805 wait complete. 2023-11-15T06:44:52.811 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.805+0000 7f3050644700 1 Processor -- start 2023-11-15T06:44:52.811 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.805+0000 7f3050644700 1 -- start start 2023-11-15T06:44:52.811 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f3050644700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f3048105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:52.812 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f3050644700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3048105b20 con 0x7f30480f5410 2023-11-15T06:44:52.812 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f304e3e0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f3048105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:52.812 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f304e3e0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f3048105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:50654/0 (socket says 172.21.15.135:50654) 2023-11-15T06:44:52.813 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f304e3e0700 1 -- 172.21.15.135:0/4128476523 learned_addr learned my addr 172.21.15.135:0/4128476523 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:52.813 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f304e3e0700 1 -- 172.21.15.135:0/4128476523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3038005040 con 0x7f30480f5410 2023-11-15T06:44:52.813 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f304e3e0700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f3048105630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f3038013510 tx=0x7f303800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:52.813 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f303effd700 1 -- 172.21.15.135:0/4128476523 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3038009320 con 0x7f30480f5410 2023-11-15T06:44:52.813 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f303effd700 1 -- 172.21.15.135:0/4128476523 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3038013f00 con 0x7f30480f5410 2023-11-15T06:44:52.814 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3048105d20 con 0x7f30480f5410 2023-11-15T06:44:52.814 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.806+0000 7f303effd700 1 -- 172.21.15.135:0/4128476523 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3038014d90 con 0x7f30480f5410 2023-11-15T06:44:52.814 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.807+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30481061b0 con 0x7f30480f5410 2023-11-15T06:44:52.814 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.807+0000 7f303effd700 1 -- 172.21.15.135:0/4128476523 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30380095f0 con 0x7f30480f5410 2023-11-15T06:44:52.814 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.807+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3048045ad0 con 0x7f30480f5410 2023-11-15T06:44:52.815 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.807+0000 7f303effd700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f303402ff60 0x7f3034032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:52.815 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.807+0000 7f303effd700 1 -- 172.21.15.135:0/4128476523 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30380410a0 con 0x7f30480f5410 2023-11-15T06:44:52.815 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.808+0000 7f304dbdf700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f303402ff60 0x7f3034032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:52.815 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.808+0000 7f304dbdf700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f303402ff60 0x7f3034032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f30440096a0 tx=0x7f3044006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:52.816 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:52.812+0000 7f303effd700 1 -- 172.21.15.135:0/4128476523 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f303801d7f0 con 0x7f30480f5410 2023-11-15T06:44:53.019 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.016+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30480fae60 con 0x7f30480f5410 2023-11-15T06:44:53.020 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.017+0000 7f303effd700 1 -- 172.21.15.135:0/4128476523 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3038010230 con 0x7f30480f5410 2023-11-15T06:44:53.020 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:53.020 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:53.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f303402ff60 msgr2=0x7f3034032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:53.021 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f303402ff60 0x7f3034032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f30440096a0 tx=0x7f3044006b40).stop 2023-11-15T06:44:53.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 msgr2=0x7f3048105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:53.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f3048105630 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f3038013510 tx=0x7f303800b040).stop 2023-11-15T06:44:53.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 shutdown_connections 2023-11-15T06:44:53.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f303402ff60 0x7f3034032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:53.022 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 --2- 172.21.15.135:0/4128476523 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f30480f5410 0x7f3048105630 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:53.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 >> 172.21.15.135:0/4128476523 conn(0x7f30480f0b70 msgr2=0x7f30480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:53.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 shutdown_connections 2023-11-15T06:44:53.023 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:53.018+0000 7f3050644700 1 -- 172.21.15.135:0/4128476523 wait complete. 2023-11-15T06:44:53.023 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:53.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:52.298614+0000 mgr.smithi123.razjtn (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:53.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:53.017874+0000 mon.smithi123 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.135:0/4128476523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:54.413 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:54.414 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:55.633 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.629+0000 7f852aba6700 1 -- 172.21.15.135:0/2025668100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 msgr2=0x7f85240f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:55.633 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.629+0000 7f852aba6700 1 --2- 172.21.15.135:0/2025668100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f85240f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7f8518009fb0 tx=0x7f8518009b30).stop 2023-11-15T06:44:55.633 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.629+0000 7f852aba6700 1 -- 172.21.15.135:0/2025668100 shutdown_connections 2023-11-15T06:44:55.633 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.629+0000 7f852aba6700 1 --2- 172.21.15.135:0/2025668100 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f85240f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:55.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.629+0000 7f852aba6700 1 -- 172.21.15.135:0/2025668100 >> 172.21.15.135:0/2025668100 conn(0x7f85240f0b70 msgr2=0x7f85240f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:55.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.629+0000 7f852aba6700 1 -- 172.21.15.135:0/2025668100 shutdown_connections 2023-11-15T06:44:55.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.629+0000 7f852aba6700 1 -- 172.21.15.135:0/2025668100 wait complete. 2023-11-15T06:44:55.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.630+0000 7f852aba6700 1 Processor -- start 2023-11-15T06:44:55.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.630+0000 7f852aba6700 1 -- start start 2023-11-15T06:44:55.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.630+0000 7f852aba6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f8524106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:55.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.630+0000 7f852aba6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8524106c50 con 0x7f85240f5410 2023-11-15T06:44:55.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f8528942700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f8524106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:55.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f8528942700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f8524106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:50674/0 (socket says 172.21.15.135:50674) 2023-11-15T06:44:55.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f8528942700 1 -- 172.21.15.135:0/2300950418 learned_addr learned my addr 172.21.15.135:0/2300950418 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:55.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f8528942700 1 -- 172.21.15.135:0/2300950418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8518005040 con 0x7f85240f5410 2023-11-15T06:44:55.637 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f8528942700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f8524106760 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f8518009f80 tx=0x7f851800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:55.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f85217fa700 1 -- 172.21.15.135:0/2300950418 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8518009320 con 0x7f85240f5410 2023-11-15T06:44:55.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8524106e50 con 0x7f85240f5410 2023-11-15T06:44:55.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f85217fa700 1 -- 172.21.15.135:0/2300950418 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8518013ec0 con 0x7f85240f5410 2023-11-15T06:44:55.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f85217fa700 1 -- 172.21.15.135:0/2300950418 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f851801ece0 con 0x7f85240f5410 2023-11-15T06:44:55.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.631+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f85241072e0 con 0x7f85240f5410 2023-11-15T06:44:55.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.632+0000 7f85217fa700 1 -- 172.21.15.135:0/2300950418 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f851801e420 con 0x7f85240f5410 2023-11-15T06:44:55.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.632+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8524045ad0 con 0x7f85240f5410 2023-11-15T06:44:55.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.632+0000 7f85217fa700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:55.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.632+0000 7f85217fa700 1 -- 172.21.15.135:0/2300950418 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8518040e10 con 0x7f85240f5410 2023-11-15T06:44:55.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.632+0000 7f8523fff700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:55.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.633+0000 7f8523fff700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f851402ff60 0x7f8514032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f85100096a0 tx=0x7f8510006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:55.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.637+0000 7f85217fa700 1 -- 172.21.15.135:0/2300950418 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f851801e6f0 con 0x7f85240f5410 2023-11-15T06:44:55.841 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.838+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8524108130 con 0x7f85240f5410 2023-11-15T06:44:55.842 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.839+0000 7f85217fa700 1 -- 172.21.15.135:0/2300950418 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f851801cb10 con 0x7f85240f5410 2023-11-15T06:44:55.843 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:55.843 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:55.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f851402ff60 msgr2=0x7f8514032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:55.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f851402ff60 0x7f8514032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f85100096a0 tx=0x7f8510006b40).stop 2023-11-15T06:44:55.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 msgr2=0x7f8524106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:55.844 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f8524106760 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7f8518009f80 tx=0x7f851800b040).stop 2023-11-15T06:44:55.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 shutdown_connections 2023-11-15T06:44:55.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f851402ff60 0x7f8514032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:55.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 --2- 172.21.15.135:0/2300950418 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f85240f5410 0x7f8524106760 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:55.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 >> 172.21.15.135:0/2300950418 conn(0x7f85240f0b70 msgr2=0x7f852405fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:55.845 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 shutdown_connections 2023-11-15T06:44:55.846 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:55.841+0000 7f852aba6700 1 -- 172.21.15.135:0/2300950418 wait complete. 2023-11-15T06:44:55.846 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:55.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:54.299033+0000 mgr.smithi123.razjtn (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:56.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:56 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:55.840447+0000 mon.smithi123 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.135:0/2300950418' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:57.222 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:57.223 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:44:57.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:56.299435+0000 mgr.smithi123.razjtn (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:58.378 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.372+0000 7f9970699700 1 -- 172.21.15.135:0/1720144295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 msgr2=0x7f99680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:58.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.372+0000 7f9970699700 1 --2- 172.21.15.135:0/1720144295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f99680f5830 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f9958009fb0 tx=0x7f9958009b30).stop 2023-11-15T06:44:58.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.372+0000 7f9970699700 1 -- 172.21.15.135:0/1720144295 shutdown_connections 2023-11-15T06:44:58.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.372+0000 7f9970699700 1 --2- 172.21.15.135:0/1720144295 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f99680f5830 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:58.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.372+0000 7f9970699700 1 -- 172.21.15.135:0/1720144295 >> 172.21.15.135:0/1720144295 conn(0x7f99680f0b70 msgr2=0x7f99680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:58.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.372+0000 7f9970699700 1 -- 172.21.15.135:0/1720144295 shutdown_connections 2023-11-15T06:44:58.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.372+0000 7f9970699700 1 -- 172.21.15.135:0/1720144295 wait complete. 2023-11-15T06:44:58.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f9970699700 1 Processor -- start 2023-11-15T06:44:58.382 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f9970699700 1 -- start start 2023-11-15T06:44:58.382 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f9970699700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f9968108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:58.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f9970699700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9968108e00 con 0x7f99680f5410 2023-11-15T06:44:58.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f996e435700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f9968108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:58.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f996e435700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f9968108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56522/0 (socket says 172.21.15.135:56522) 2023-11-15T06:44:58.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f996e435700 1 -- 172.21.15.135:0/1432261596 learned_addr learned my addr 172.21.15.135:0/1432261596 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:44:58.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.373+0000 7f996e435700 1 -- 172.21.15.135:0/1432261596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9958005040 con 0x7f99680f5410 2023-11-15T06:44:58.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.374+0000 7f996e435700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f9968108910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9958014000 tx=0x7f995800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:58.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.374+0000 7f995effd700 1 -- 172.21.15.135:0/1432261596 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99580095f0 con 0x7f99680f5410 2023-11-15T06:44:58.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.374+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9968109000 con 0x7f99680f5410 2023-11-15T06:44:58.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.374+0000 7f995effd700 1 -- 172.21.15.135:0/1432261596 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9958007e70 con 0x7f99680f5410 2023-11-15T06:44:58.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.374+0000 7f995effd700 1 -- 172.21.15.135:0/1432261596 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9958014ea0 con 0x7f99680f5410 2023-11-15T06:44:58.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.374+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9968109490 con 0x7f99680f5410 2023-11-15T06:44:58.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.375+0000 7f995effd700 1 -- 172.21.15.135:0/1432261596 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9958025070 con 0x7f99680f5410 2023-11-15T06:44:58.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.375+0000 7f995effd700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f995402ff60 0x7f9954032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:44:58.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.375+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9968045ad0 con 0x7f99680f5410 2023-11-15T06:44:58.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.375+0000 7f995effd700 1 -- 172.21.15.135:0/1432261596 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9958040990 con 0x7f99680f5410 2023-11-15T06:44:58.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.375+0000 7f996dc34700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f995402ff60 0x7f9954032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:44:58.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.375+0000 7f996dc34700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f995402ff60 0x7f9954032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f99640096a0 tx=0x7f9964006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:44:58.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.379+0000 7f995effd700 1 -- 172.21.15.135:0/1432261596 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9958026350 con 0x7f99680f5410 2023-11-15T06:44:58.587 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.584+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99680faf50 con 0x7f99680f5410 2023-11-15T06:44:58.588 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.585+0000 7f995effd700 1 -- 172.21.15.135:0/1432261596 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9958012330 con 0x7f99680f5410 2023-11-15T06:44:58.589 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:44:58.589 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:44:58.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f995402ff60 msgr2=0x7f9954032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:58.590 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f995402ff60 0x7f9954032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f99640096a0 tx=0x7f9964006b40).stop 2023-11-15T06:44:58.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 msgr2=0x7f9968108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:44:58.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f9968108910 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f9958014000 tx=0x7f995800b040).stop 2023-11-15T06:44:58.591 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 shutdown_connections 2023-11-15T06:44:58.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f995402ff60 0x7f9954032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:58.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 --2- 172.21.15.135:0/1432261596 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f99680f5410 0x7f9968108910 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:44:58.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 >> 172.21.15.135:0/1432261596 conn(0x7f99680f0b70 msgr2=0x7f996805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:44:58.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 shutdown_connections 2023-11-15T06:44:58.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:44:58.587+0000 7f9970699700 1 -- 172.21.15.135:0/1432261596 wait complete. 2023-11-15T06:44:58.594 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:44:58.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:58 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:44:58.586354+0000 mon.smithi123 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.135:0/1432261596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:44:59.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:44:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:44:58.299865+0000 mgr.smithi123.razjtn (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:44:59.971 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:44:59.972 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:01.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.141+0000 7f8792a1a700 1 -- 172.21.15.135:0/801020277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 msgr2=0x7f878c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:01.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.141+0000 7f8792a1a700 1 --2- 172.21.15.135:0/801020277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c0f6930 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7f877c009fb0 tx=0x7f877c009b30).stop 2023-11-15T06:45:01.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.141+0000 7f8792a1a700 1 -- 172.21.15.135:0/801020277 shutdown_connections 2023-11-15T06:45:01.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.141+0000 7f8792a1a700 1 --2- 172.21.15.135:0/801020277 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c0f6930 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:01.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.141+0000 7f8792a1a700 1 -- 172.21.15.135:0/801020277 >> 172.21.15.135:0/801020277 conn(0x7f878c0f1c70 msgr2=0x7f878c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:01.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.141+0000 7f8792a1a700 1 -- 172.21.15.135:0/801020277 shutdown_connections 2023-11-15T06:45:01.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.141+0000 7f8792a1a700 1 -- 172.21.15.135:0/801020277 wait complete. 2023-11-15T06:45:01.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.142+0000 7f8792a1a700 1 Processor -- start 2023-11-15T06:45:01.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.142+0000 7f8792a1a700 1 -- start start 2023-11-15T06:45:01.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f8792a1a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:01.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f8792a1a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f878c106c20 con 0x7f878c0f6510 2023-11-15T06:45:01.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f878bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:01.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f878bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56534/0 (socket says 172.21.15.135:56534) 2023-11-15T06:45:01.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f878bfff700 1 -- 172.21.15.135:0/2048071976 learned_addr learned my addr 172.21.15.135:0/2048071976 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:01.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f878bfff700 1 -- 172.21.15.135:0/2048071976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f877c005040 con 0x7f878c0f6510 2023-11-15T06:45:01.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f878bfff700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f877c013510 tx=0x7f877c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:01.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f8788ff9700 1 -- 172.21.15.135:0/2048071976 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f877c009320 con 0x7f878c0f6510 2023-11-15T06:45:01.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f878c106e20 con 0x7f878c0f6510 2023-11-15T06:45:01.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f8788ff9700 1 -- 172.21.15.135:0/2048071976 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f877c013f00 con 0x7f878c0f6510 2023-11-15T06:45:01.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f8788ff9700 1 -- 172.21.15.135:0/2048071976 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f877c014d90 con 0x7f878c0f6510 2023-11-15T06:45:01.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.143+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f878c1072b0 con 0x7f878c0f6510 2023-11-15T06:45:01.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.144+0000 7f8788ff9700 1 -- 172.21.15.135:0/2048071976 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f877c0263a0 con 0x7f878c0f6510 2023-11-15T06:45:01.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.144+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f878c045ad0 con 0x7f878c0f6510 2023-11-15T06:45:01.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.144+0000 7f8788ff9700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:01.152 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.144+0000 7f8788ff9700 1 -- 172.21.15.135:0/2048071976 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f877c0412e0 con 0x7f878c0f6510 2023-11-15T06:45:01.152 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.145+0000 7f878b7fe700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:01.152 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.145+0000 7f878b7fe700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f877802ff60 0x7f8778032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f87740096a0 tx=0x7f8774006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:01.153 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.149+0000 7f8788ff9700 1 -- 172.21.15.135:0/2048071976 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f877c0126f0 con 0x7f878c0f6510 2023-11-15T06:45:01.356 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.354+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f878c0fbf60 con 0x7f878c0f6510 2023-11-15T06:45:01.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.355+0000 7f8788ff9700 1 -- 172.21.15.135:0/2048071976 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f877c012330 con 0x7f878c0f6510 2023-11-15T06:45:01.358 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:01.359 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:01.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f877802ff60 msgr2=0x7f8778032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:01.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f877802ff60 0x7f8778032420 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f87740096a0 tx=0x7f8774006b40).stop 2023-11-15T06:45:01.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 msgr2=0x7f878c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:01.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c106730 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7f877c013510 tx=0x7f877c00b040).stop 2023-11-15T06:45:01.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 shutdown_connections 2023-11-15T06:45:01.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f877802ff60 0x7f8778032420 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:01.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 --2- 172.21.15.135:0/2048071976 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f878c0f6510 0x7f878c106730 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:01.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 >> 172.21.15.135:0/2048071976 conn(0x7f878c0f1c70 msgr2=0x7f878c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:01.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 shutdown_connections 2023-11-15T06:45:01.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:01.356+0000 7f8792a1a700 1 -- 172.21.15.135:0/2048071976 wait complete. 2023-11-15T06:45:01.362 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:01.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:00.300167+0000 mgr.smithi123.razjtn (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:01.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:01.355661+0000 mon.smithi123 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.135:0/2048071976' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:02.746 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:02.773 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:03.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:02.300528+0000 mgr.smithi123.razjtn (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:03.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.964+0000 7f2fd9519700 1 -- 172.21.15.135:0/3159768832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 msgr2=0x7f2fd40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:03.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.964+0000 7f2fd9519700 1 --2- 172.21.15.135:0/3159768832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd40f90f0 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f2fbc009fb0 tx=0x7f2fbc009b30).stop 2023-11-15T06:45:03.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.964+0000 7f2fd9519700 1 -- 172.21.15.135:0/3159768832 shutdown_connections 2023-11-15T06:45:03.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.964+0000 7f2fd9519700 1 --2- 172.21.15.135:0/3159768832 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd40f90f0 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:03.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.964+0000 7f2fd9519700 1 -- 172.21.15.135:0/3159768832 >> 172.21.15.135:0/3159768832 conn(0x7f2fd40f0b70 msgr2=0x7f2fd40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:03.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.965+0000 7f2fd9519700 1 -- 172.21.15.135:0/3159768832 shutdown_connections 2023-11-15T06:45:03.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.965+0000 7f2fd9519700 1 -- 172.21.15.135:0/3159768832 wait complete. 2023-11-15T06:45:03.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.965+0000 7f2fd9519700 1 Processor -- start 2023-11-15T06:45:03.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.965+0000 7f2fd9519700 1 -- start start 2023-11-15T06:45:03.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.965+0000 7f2fd9519700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd41066c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:03.971 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.965+0000 7f2fd9519700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2fd4106bb0 con 0x7f2fd4060280 2023-11-15T06:45:03.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fd2ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd41066c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:03.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fd2ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd41066c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56550/0 (socket says 172.21.15.135:56550) 2023-11-15T06:45:03.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fd2ffd700 1 -- 172.21.15.135:0/3981377076 learned_addr learned my addr 172.21.15.135:0/3981377076 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:03.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fd2ffd700 1 -- 172.21.15.135:0/3981377076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2fbc005040 con 0x7f2fd4060280 2023-11-15T06:45:03.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fd2ffd700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd41066c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2fbc013510 tx=0x7f2fbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:03.972 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fcb7fe700 1 -- 172.21.15.135:0/3981377076 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fbc009320 con 0x7f2fd4060280 2023-11-15T06:45:03.973 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fcb7fe700 1 -- 172.21.15.135:0/3981377076 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2fbc013f00 con 0x7f2fd4060280 2023-11-15T06:45:03.973 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2fd4106db0 con 0x7f2fd4060280 2023-11-15T06:45:03.973 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fcb7fe700 1 -- 172.21.15.135:0/3981377076 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2fbc014dc0 con 0x7f2fd4060280 2023-11-15T06:45:03.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.966+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2fd4107240 con 0x7f2fd4060280 2023-11-15T06:45:03.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.967+0000 7f2fcb7fe700 1 -- 172.21.15.135:0/3981377076 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2fbc014420 con 0x7f2fd4060280 2023-11-15T06:45:03.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.967+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2fd4045ad0 con 0x7f2fd4060280 2023-11-15T06:45:03.974 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.967+0000 7f2fcb7fe700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2fc002ff60 0x7f2fc0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:03.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.967+0000 7f2fcb7fe700 1 -- 172.21.15.135:0/3981377076 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2fbc041030 con 0x7f2fd4060280 2023-11-15T06:45:03.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.967+0000 7f2fd27fc700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2fc002ff60 0x7f2fc0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:03.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.968+0000 7f2fd27fc700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2fc002ff60 0x7f2fc0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f2fc40096a0 tx=0x7f2fc4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:03.975 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:03.972+0000 7f2fcb7fe700 1 -- 172.21.15.135:0/3981377076 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2fbc01cb10 con 0x7f2fd4060280 2023-11-15T06:45:04.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.178+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2fd41080f0 con 0x7f2fd4060280 2023-11-15T06:45:04.183 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.179+0000 7f2fcb7fe700 1 -- 172.21.15.135:0/3981377076 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2fbc01cb10 con 0x7f2fd4060280 2023-11-15T06:45:04.183 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:04.183 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:04.184 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2fc002ff60 msgr2=0x7f2fc0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:04.184 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2fc002ff60 0x7f2fc0032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f2fc40096a0 tx=0x7f2fc4006b40).stop 2023-11-15T06:45:04.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 msgr2=0x7f2fd41066c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:04.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd41066c0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f2fbc013510 tx=0x7f2fbc00b040).stop 2023-11-15T06:45:04.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 shutdown_connections 2023-11-15T06:45:04.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f2fc002ff60 0x7f2fc0032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:04.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 --2- 172.21.15.135:0/3981377076 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f2fd4060280 0x7f2fd41066c0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:04.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 >> 172.21.15.135:0/3981377076 conn(0x7f2fd40f0b70 msgr2=0x7f2fd40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:04.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 shutdown_connections 2023-11-15T06:45:04.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:04.181+0000 7f2fd9519700 1 -- 172.21.15.135:0/3981377076 wait complete. 2023-11-15T06:45:04.186 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:04.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:04 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:04.180170+0000 mon.smithi123 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.135:0/3981377076' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:05.564 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:05.564 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:05.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:04.300865+0000 mgr.smithi123.razjtn (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:06.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.673+0000 7f7d9275c700 1 -- 172.21.15.135:0/1354835482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 msgr2=0x7f7d8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:06.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.673+0000 7f7d9275c700 1 --2- 172.21.15.135:0/1354835482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c0f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f7d7c009fb0 tx=0x7f7d7c009b30).stop 2023-11-15T06:45:06.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.673+0000 7f7d9275c700 1 -- 172.21.15.135:0/1354835482 shutdown_connections 2023-11-15T06:45:06.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.673+0000 7f7d9275c700 1 --2- 172.21.15.135:0/1354835482 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c0f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:06.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.673+0000 7f7d9275c700 1 -- 172.21.15.135:0/1354835482 >> 172.21.15.135:0/1354835482 conn(0x7f7d8c0f0b70 msgr2=0x7f7d8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:06.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.673+0000 7f7d9275c700 1 -- 172.21.15.135:0/1354835482 shutdown_connections 2023-11-15T06:45:06.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.673+0000 7f7d9275c700 1 -- 172.21.15.135:0/1354835482 wait complete. 2023-11-15T06:45:06.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.674+0000 7f7d9275c700 1 Processor -- start 2023-11-15T06:45:06.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.674+0000 7f7d9275c700 1 -- start start 2023-11-15T06:45:06.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.674+0000 7f7d9275c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:06.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.674+0000 7f7d9275c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7d8c060240 con 0x7f7d8c0f5410 2023-11-15T06:45:06.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d8bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:06.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d8bfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:56566/0 (socket says 172.21.15.135:56566) 2023-11-15T06:45:06.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d8bfff700 1 -- 172.21.15.135:0/782202039 learned_addr learned my addr 172.21.15.135:0/782202039 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:06.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d8bfff700 1 -- 172.21.15.135:0/782202039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7d7c005040 con 0x7f7d8c0f5410 2023-11-15T06:45:06.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d8bfff700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c109430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7d7c013510 tx=0x7f7d7c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:06.681 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d88ff9700 1 -- 172.21.15.135:0/782202039 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d7c009320 con 0x7f7d8c0f5410 2023-11-15T06:45:06.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d88ff9700 1 -- 172.21.15.135:0/782202039 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7d7c013f00 con 0x7f7d8c0f5410 2023-11-15T06:45:06.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7d8c060440 con 0x7f7d8c0f5410 2023-11-15T06:45:06.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d88ff9700 1 -- 172.21.15.135:0/782202039 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7d7c014d90 con 0x7f7d8c0f5410 2023-11-15T06:45:06.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.675+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7d8c109ae0 con 0x7f7d8c0f5410 2023-11-15T06:45:06.682 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.676+0000 7f7d88ff9700 1 -- 172.21.15.135:0/782202039 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7d7c014420 con 0x7f7d8c0f5410 2023-11-15T06:45:06.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.676+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7d8c045ad0 con 0x7f7d8c0f5410 2023-11-15T06:45:06.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.676+0000 7f7d88ff9700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d7402ff60 0x7f7d74032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:06.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.676+0000 7f7d88ff9700 1 -- 172.21.15.135:0/782202039 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7d7c0410c0 con 0x7f7d8c0f5410 2023-11-15T06:45:06.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.676+0000 7f7d8b7fe700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d7402ff60 0x7f7d74032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:06.683 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.677+0000 7f7d8b7fe700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d7402ff60 0x7f7d74032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7d800096a0 tx=0x7f7d80006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:06.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.681+0000 7f7d88ff9700 1 -- 172.21.15.135:0/782202039 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7d7c0095f0 con 0x7f7d8c0f5410 2023-11-15T06:45:06.890 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.888+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7d8c0fae60 con 0x7f7d8c0f5410 2023-11-15T06:45:06.891 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.888+0000 7f7d88ff9700 1 -- 172.21.15.135:0/782202039 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7d7c01db70 con 0x7f7d8c0f5410 2023-11-15T06:45:06.891 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:06.892 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:06.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d7402ff60 msgr2=0x7f7d74032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:06.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d7402ff60 0x7f7d74032420 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f7d800096a0 tx=0x7f7d80006b40).stop 2023-11-15T06:45:06.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 msgr2=0x7f7d8c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:06.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c109430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f7d7c013510 tx=0x7f7d7c00b040).stop 2023-11-15T06:45:06.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 shutdown_connections 2023-11-15T06:45:06.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f7d7402ff60 0x7f7d74032420 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:06.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 --2- 172.21.15.135:0/782202039 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f7d8c0f5410 0x7f7d8c109430 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:06.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 >> 172.21.15.135:0/782202039 conn(0x7f7d8c0f0b70 msgr2=0x7f7d8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:06.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 shutdown_connections 2023-11-15T06:45:06.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:06.890+0000 7f7d9275c700 1 -- 172.21.15.135:0/782202039 wait complete. 2023-11-15T06:45:06.896 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:07.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:06.301179+0000 mgr.smithi123.razjtn (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:07.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:06.889473+0000 mon.smithi123 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.135:0/782202039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:07.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:45:06.945158+0000 mgr.smithi123.razjtn (mgr.14162) 230 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi123 ... 2023-11-15T06:45:07.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Verifying port 9100 ... 2023-11-15T06:45:07.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Non-zero exit code 1 from systemctl start ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123 2023-11-15T06:45:07.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: systemctl: stderr Job for ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123.service failed because a timeout was exceeded. 2023-11-15T06:45:07.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: systemctl: stderr See "systemctl status ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123.service" and "journalctl -xe" for details. 2023-11-15T06:45:07.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Traceback (most recent call last): 2023-11-15T06:45:07.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:45:07.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: main() 2023-11-15T06:45:07.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:45:07.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: r = ctx.func(ctx) 2023-11-15T06:45:07.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:45:07.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: return func(ctx) 2023-11-15T06:45:07.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T06:45:07.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ports=daemon_ports) 2023-11-15T06:45:07.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T06:45:07.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T06:45:07.910 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T06:45:07.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T06:45:07.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T06:45:07.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T06:45:07.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: RuntimeError: Failed command: systemctl start ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123 2023-11-15T06:45:07.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Traceback (most recent call last): 2023-11-15T06:45:07.911 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2023-11-15T06:45:07.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: yield (conn, connr) 2023-11-15T06:45:07.912 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2023-11-15T06:45:07.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: code, '\n'.join(err))) 2023-11-15T06:45:07.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi123 ... 2023-11-15T06:45:07.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Verifying port 9100 ... 2023-11-15T06:45:07.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Non-zero exit code 1 from systemctl start ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123 2023-11-15T06:45:07.913 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: systemctl: stderr Job for ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123.service failed because a timeout was exceeded. 2023-11-15T06:45:07.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: systemctl: stderr See "systemctl status ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123.service" and "journalctl -xe" for details. 2023-11-15T06:45:07.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: Traceback (most recent call last): 2023-11-15T06:45:07.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2023-11-15T06:45:07.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: main() 2023-11-15T06:45:07.914 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2023-11-15T06:45:07.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: r = ctx.func(ctx) 2023-11-15T06:45:07.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2023-11-15T06:45:07.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: return func(ctx) 2023-11-15T06:45:07.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2023-11-15T06:45:07.915 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: ports=daemon_ports) 2023-11-15T06:45:07.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2023-11-15T06:45:07.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: c, osd_fsid=osd_fsid, ports=ports) 2023-11-15T06:45:07.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2023-11-15T06:45:07.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2023-11-15T06:45:07.916 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: File "/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2023-11-15T06:45:07.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2023-11-15T06:45:07.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: RuntimeError: Failed command: systemctl start ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@node-exporter.smithi123 2023-11-15T06:45:07.917 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cephadm 2023-11-15T06:45:06.952511+0000 mgr.smithi123.razjtn (mgr.14162) 231 : cephadm [INF] Deploying daemon prometheus.smithi123 on smithi123 2023-11-15T06:45:08.242 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:08.243 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:09.355 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.350+0000 7f84c2527700 1 -- 172.21.15.135:0/1656760220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 msgr2=0x7f84bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:09.355 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.350+0000 7f84c2527700 1 --2- 172.21.15.135:0/1656760220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc0f6930 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f84ac009fb0 tx=0x7f84ac009b30).stop 2023-11-15T06:45:09.356 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.350+0000 7f84c2527700 1 -- 172.21.15.135:0/1656760220 shutdown_connections 2023-11-15T06:45:09.356 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.350+0000 7f84c2527700 1 --2- 172.21.15.135:0/1656760220 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc0f6930 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:09.356 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.350+0000 7f84c2527700 1 -- 172.21.15.135:0/1656760220 >> 172.21.15.135:0/1656760220 conn(0x7f84bc0f1c70 msgr2=0x7f84bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:09.356 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.350+0000 7f84c2527700 1 -- 172.21.15.135:0/1656760220 shutdown_connections 2023-11-15T06:45:09.356 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.350+0000 7f84c2527700 1 -- 172.21.15.135:0/1656760220 wait complete. 2023-11-15T06:45:09.357 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.351+0000 7f84c2527700 1 Processor -- start 2023-11-15T06:45:09.357 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.351+0000 7f84c2527700 1 -- start start 2023-11-15T06:45:09.357 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.351+0000 7f84c2527700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:09.357 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.351+0000 7f84c2527700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84bc106c20 con 0x7f84bc0f6510 2023-11-15T06:45:09.357 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.351+0000 7f84bbfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:09.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.351+0000 7f84bbfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:41258/0 (socket says 172.21.15.135:41258) 2023-11-15T06:45:09.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.351+0000 7f84bbfff700 1 -- 172.21.15.135:0/2439767641 learned_addr learned my addr 172.21.15.135:0/2439767641 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:09.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.352+0000 7f84bbfff700 1 -- 172.21.15.135:0/2439767641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f84ac005040 con 0x7f84bc0f6510 2023-11-15T06:45:09.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.352+0000 7f84bbfff700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc106730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f84ac013510 tx=0x7f84ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:09.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.352+0000 7f84b8ff9700 1 -- 172.21.15.135:0/2439767641 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ac009320 con 0x7f84bc0f6510 2023-11-15T06:45:09.359 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.352+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84bc106e20 con 0x7f84bc0f6510 2023-11-15T06:45:09.359 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.352+0000 7f84b8ff9700 1 -- 172.21.15.135:0/2439767641 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f84ac013f00 con 0x7f84bc0f6510 2023-11-15T06:45:09.359 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.352+0000 7f84b8ff9700 1 -- 172.21.15.135:0/2439767641 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f84ac014d90 con 0x7f84bc0f6510 2023-11-15T06:45:09.359 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.352+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84bc1072b0 con 0x7f84bc0f6510 2023-11-15T06:45:09.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.353+0000 7f84b8ff9700 1 -- 172.21.15.135:0/2439767641 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f84ac0095f0 con 0x7f84bc0f6510 2023-11-15T06:45:09.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.353+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84bc045ad0 con 0x7f84bc0f6510 2023-11-15T06:45:09.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.353+0000 7f84b8ff9700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f84a802ff60 0x7f84a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:09.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.353+0000 7f84b8ff9700 1 -- 172.21.15.135:0/2439767641 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f84ac041000 con 0x7f84bc0f6510 2023-11-15T06:45:09.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.353+0000 7f84bb7fe700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f84a802ff60 0x7f84a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:09.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.354+0000 7f84bb7fe700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f84a802ff60 0x7f84a8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f84a40096a0 tx=0x7f84a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:09.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.358+0000 7f84b8ff9700 1 -- 172.21.15.135:0/2439767641 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f84ac01d7f0 con 0x7f84bc0f6510 2023-11-15T06:45:09.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.566+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84bc0fbf60 con 0x7f84bc0f6510 2023-11-15T06:45:09.569 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.567+0000 7f84b8ff9700 1 -- 172.21.15.135:0/2439767641 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f84ac010230 con 0x7f84bc0f6510 2023-11-15T06:45:09.570 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:09.570 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:09.571 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f84a802ff60 msgr2=0x7f84a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:09.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f84a802ff60 0x7f84a8032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f84a40096a0 tx=0x7f84a4006b40).stop 2023-11-15T06:45:09.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 msgr2=0x7f84bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:09.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc106730 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f84ac013510 tx=0x7f84ac00b040).stop 2023-11-15T06:45:09.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 shutdown_connections 2023-11-15T06:45:09.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f84a802ff60 0x7f84a8032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:09.572 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 --2- 172.21.15.135:0/2439767641 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84bc0f6510 0x7f84bc106730 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:09.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 >> 172.21.15.135:0/2439767641 conn(0x7f84bc0f1c70 msgr2=0x7f84bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:09.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.568+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 shutdown_connections 2023-11-15T06:45:09.573 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:09.569+0000 7f84c2527700 1 -- 172.21.15.135:0/2439767641 wait complete. 2023-11-15T06:45:09.574 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:08.301526+0000 mgr.smithi123.razjtn (mgr.14162) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:09.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:09.567829+0000 mon.smithi123 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.135:0/2439767641' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:10.950 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:10.951 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:11.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:10.301890+0000 mgr.smithi123.razjtn (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:12.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 -- 172.21.15.135:0/1272070556 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 msgr2=0x7fb02c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:12.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 --2- 172.21.15.135:0/1272070556 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c0f66c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7fb028009fb0 tx=0x7fb028009b30).stop 2023-11-15T06:45:12.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 -- 172.21.15.135:0/1272070556 shutdown_connections 2023-11-15T06:45:12.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 --2- 172.21.15.135:0/1272070556 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c0f66c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:12.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 -- 172.21.15.135:0/1272070556 >> 172.21.15.135:0/1272070556 conn(0x7fb02c0f1aa0 msgr2=0x7fb02c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:12.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 -- 172.21.15.135:0/1272070556 shutdown_connections 2023-11-15T06:45:12.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 -- 172.21.15.135:0/1272070556 wait complete. 2023-11-15T06:45:12.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 Processor -- start 2023-11-15T06:45:12.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.106+0000 7fb032ee5700 1 -- start start 2023-11-15T06:45:12.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb032ee5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:12.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb032ee5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb02c1069a0 con 0x7fb02c0f62a0 2023-11-15T06:45:12.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb031ee3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:12.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb031ee3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:41268/0 (socket says 172.21.15.135:41268) 2023-11-15T06:45:12.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb031ee3700 1 -- 172.21.15.135:0/68628872 learned_addr learned my addr 172.21.15.135:0/68628872 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:12.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb031ee3700 1 -- 172.21.15.135:0/68628872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb028005040 con 0x7fb02c0f62a0 2023-11-15T06:45:12.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb031ee3700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c1064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fb028013510 tx=0x7fb02800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:12.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb0227fc700 1 -- 172.21.15.135:0/68628872 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb028009320 con 0x7fb02c0f62a0 2023-11-15T06:45:12.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb02c106ba0 con 0x7fb02c0f62a0 2023-11-15T06:45:12.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb0227fc700 1 -- 172.21.15.135:0/68628872 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb028013f00 con 0x7fb02c0f62a0 2023-11-15T06:45:12.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb0227fc700 1 -- 172.21.15.135:0/68628872 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb028014dc0 con 0x7fb02c0f62a0 2023-11-15T06:45:12.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.107+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb02c107030 con 0x7fb02c0f62a0 2023-11-15T06:45:12.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.108+0000 7fb0227fc700 1 -- 172.21.15.135:0/68628872 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb0280095f0 con 0x7fb02c0f62a0 2023-11-15T06:45:12.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.108+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb02c045ad0 con 0x7fb02c0f62a0 2023-11-15T06:45:12.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.108+0000 7fb0227fc700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb01802ff60 0x7fb018032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:12.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.108+0000 7fb0227fc700 1 -- 172.21.15.135:0/68628872 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb028041100 con 0x7fb02c0f62a0 2023-11-15T06:45:12.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.109+0000 7fb0316e2700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb01802ff60 0x7fb018032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:12.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.109+0000 7fb0316e2700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb01802ff60 0x7fb018032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb01c0096a0 tx=0x7fb01c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:12.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.113+0000 7fb0227fc700 1 -- 172.21.15.135:0/68628872 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb028014420 con 0x7fb02c0f62a0 2023-11-15T06:45:12.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.322+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb02c107ee0 con 0x7fb02c0f62a0 2023-11-15T06:45:12.326 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.323+0000 7fb0227fc700 1 -- 172.21.15.135:0/68628872 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb028010230 con 0x7fb02c0f62a0 2023-11-15T06:45:12.326 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:12.326 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:12.330 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb01802ff60 msgr2=0x7fb018032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:12.330 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb01802ff60 0x7fb018032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7fb01c0096a0 tx=0x7fb01c006b40).stop 2023-11-15T06:45:12.330 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 msgr2=0x7fb02c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:12.331 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c1064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7fb028013510 tx=0x7fb02800b040).stop 2023-11-15T06:45:12.331 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 shutdown_connections 2023-11-15T06:45:12.331 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb01802ff60 0x7fb018032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:12.331 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 --2- 172.21.15.135:0/68628872 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb02c0f62a0 0x7fb02c1064b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:12.331 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 >> 172.21.15.135:0/68628872 conn(0x7fb02c0f1aa0 msgr2=0x7fb02c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:12.332 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 shutdown_connections 2023-11-15T06:45:12.332 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:12.324+0000 7fb032ee5700 1 -- 172.21.15.135:0/68628872 wait complete. 2023-11-15T06:45:12.332 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:12.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:12 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:12.323858+0000 mon.smithi123 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.135:0/68628872' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:13.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:13.710 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:13.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:12.302207+0000 mgr.smithi123.razjtn (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:14.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 -- 172.21.15.135:0/551299358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 msgr2=0x7f66cc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:14.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 --2- 172.21.15.135:0/551299358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc0f55c0 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7f66bc009fb0 tx=0x7f66bc009b30).stop 2023-11-15T06:45:14.893 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 -- 172.21.15.135:0/551299358 shutdown_connections 2023-11-15T06:45:14.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 --2- 172.21.15.135:0/551299358 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc0f55c0 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:14.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 -- 172.21.15.135:0/551299358 >> 172.21.15.135:0/551299358 conn(0x7f66cc0f09a0 msgr2=0x7f66cc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:14.894 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 -- 172.21.15.135:0/551299358 shutdown_connections 2023-11-15T06:45:14.895 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 -- 172.21.15.135:0/551299358 wait complete. 2023-11-15T06:45:14.896 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 Processor -- start 2023-11-15T06:45:14.896 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 -- start start 2023-11-15T06:45:14.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:14.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.890+0000 7f66cbfff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66cc1069d0 con 0x7f66cc0f51a0 2023-11-15T06:45:14.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66caffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:14.897 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66caffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:41290/0 (socket says 172.21.15.135:41290) 2023-11-15T06:45:14.898 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66caffd700 1 -- 172.21.15.135:0/439452006 learned_addr learned my addr 172.21.15.135:0/439452006 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:14.898 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66caffd700 1 -- 172.21.15.135:0/439452006 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66bc005040 con 0x7f66cc0f51a0 2023-11-15T06:45:14.898 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66caffd700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc1064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f66bc009830 tx=0x7f66bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:14.898 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66b37fe700 1 -- 172.21.15.135:0/439452006 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66bc009320 con 0x7f66cc0f51a0 2023-11-15T06:45:14.898 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66cc106bd0 con 0x7f66cc0f51a0 2023-11-15T06:45:14.899 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66b37fe700 1 -- 172.21.15.135:0/439452006 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66bc0136c0 con 0x7f66cc0f51a0 2023-11-15T06:45:14.899 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66b37fe700 1 -- 172.21.15.135:0/439452006 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66bc01ee90 con 0x7f66cc0f51a0 2023-11-15T06:45:14.899 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.891+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66cc107060 con 0x7f66cc0f51a0 2023-11-15T06:45:14.899 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.892+0000 7f66b37fe700 1 -- 172.21.15.135:0/439452006 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66bc01e420 con 0x7f66cc0f51a0 2023-11-15T06:45:14.899 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.892+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66cc045ad0 con 0x7f66cc0f51a0 2023-11-15T06:45:14.900 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.892+0000 7f66b37fe700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f66b402ff10 0x7f66b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:14.900 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.892+0000 7f66b37fe700 1 -- 172.21.15.135:0/439452006 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66bc040fa0 con 0x7f66cc0f51a0 2023-11-15T06:45:14.900 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.893+0000 7f66ca7fc700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f66b402ff10 0x7f66b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:14.900 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.893+0000 7f66ca7fc700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f66b402ff10 0x7f66b40323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f66c00096a0 tx=0x7f66c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:14.901 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:14.897+0000 7f66b37fe700 1 -- 172.21.15.135:0/439452006 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66bc06d2b0 con 0x7f66cc0f51a0 2023-11-15T06:45:15.105 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.102+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66cc107f10 con 0x7f66cc0f51a0 2023-11-15T06:45:15.106 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.103+0000 7f66b37fe700 1 -- 172.21.15.135:0/439452006 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f66bc02c2d0 con 0x7f66cc0f51a0 2023-11-15T06:45:15.106 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:15.107 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:15.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f66b402ff10 msgr2=0x7f66b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:15.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f66b402ff10 0x7f66b40323d0 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f66c00096a0 tx=0x7f66c0006b40).stop 2023-11-15T06:45:15.108 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 msgr2=0x7f66cc1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:15.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc1064e0 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7f66bc009830 tx=0x7f66bc00b040).stop 2023-11-15T06:45:15.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 shutdown_connections 2023-11-15T06:45:15.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f66b402ff10 0x7f66b40323d0 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:15.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 --2- 172.21.15.135:0/439452006 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f66cc0f51a0 0x7f66cc1064e0 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:15.109 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 >> 172.21.15.135:0/439452006 conn(0x7f66cc0f09a0 msgr2=0x7f66cc05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:15.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 shutdown_connections 2023-11-15T06:45:15.110 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:15.105+0000 7f66cbfff700 1 -- 172.21.15.135:0/439452006 wait complete. 2023-11-15T06:45:15.110 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:15.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:14.302461+0000 mgr.smithi123.razjtn (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:15.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:15.104347+0000 mon.smithi123 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.135:0/439452006' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:16.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:16.479 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:17.592 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 -- 172.21.15.135:0/4007456051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 msgr2=0x7f46840f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:17.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 --2- 172.21.15.135:0/4007456051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46840f66c0 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f4674009fb0 tx=0x7f4674009b30).stop 2023-11-15T06:45:17.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 -- 172.21.15.135:0/4007456051 shutdown_connections 2023-11-15T06:45:17.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 --2- 172.21.15.135:0/4007456051 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46840f66c0 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:17.593 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 -- 172.21.15.135:0/4007456051 >> 172.21.15.135:0/4007456051 conn(0x7f46840f1aa0 msgr2=0x7f46840f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:17.594 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 -- 172.21.15.135:0/4007456051 shutdown_connections 2023-11-15T06:45:17.594 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 -- 172.21.15.135:0/4007456051 wait complete. 2023-11-15T06:45:17.596 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.590+0000 7f4683fff700 1 Processor -- start 2023-11-15T06:45:17.596 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4683fff700 1 -- start start 2023-11-15T06:45:17.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4683fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46841064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:17.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4683fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f46841069a0 con 0x7f46840f62a0 2023-11-15T06:45:17.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4682ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46841064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:17.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4682ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46841064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:41310/0 (socket says 172.21.15.135:41310) 2023-11-15T06:45:17.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4682ffd700 1 -- 172.21.15.135:0/615537466 learned_addr learned my addr 172.21.15.135:0/615537466 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:17.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4682ffd700 1 -- 172.21.15.135:0/615537466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4674005040 con 0x7f46840f62a0 2023-11-15T06:45:17.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.591+0000 7f4682ffd700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46841064b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4674009830 tx=0x7f467400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:17.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.592+0000 7f466b7fe700 1 -- 172.21.15.135:0/615537466 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4674009320 con 0x7f46840f62a0 2023-11-15T06:45:17.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.592+0000 7f466b7fe700 1 -- 172.21.15.135:0/615537466 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f46740136c0 con 0x7f46840f62a0 2023-11-15T06:45:17.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.592+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4684106ba0 con 0x7f46840f62a0 2023-11-15T06:45:17.599 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.592+0000 7f466b7fe700 1 -- 172.21.15.135:0/615537466 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f467401ee90 con 0x7f46840f62a0 2023-11-15T06:45:17.599 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.592+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4684107030 con 0x7f46840f62a0 2023-11-15T06:45:17.599 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.593+0000 7f466b7fe700 1 -- 172.21.15.135:0/615537466 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f467401e420 con 0x7f46840f62a0 2023-11-15T06:45:17.599 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.593+0000 7f466b7fe700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f466c02ff10 0x7f466c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:17.599 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.593+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4684045ad0 con 0x7f46840f62a0 2023-11-15T06:45:17.600 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.593+0000 7f466b7fe700 1 -- 172.21.15.135:0/615537466 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4674040ee0 con 0x7f46840f62a0 2023-11-15T06:45:17.600 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.593+0000 7f46827fc700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f466c02ff10 0x7f466c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:17.602 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.593+0000 7f46827fc700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f466c02ff10 0x7f466c0323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f46780096a0 tx=0x7f4678006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:17.603 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.597+0000 7f466b7fe700 1 -- 172.21.15.135:0/615537466 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f46740095f0 con 0x7f46840f62a0 2023-11-15T06:45:17.807 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.804+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4684107ee0 con 0x7f46840f62a0 2023-11-15T06:45:17.808 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.805+0000 7f466b7fe700 1 -- 172.21.15.135:0/615537466 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f467401c930 con 0x7f46840f62a0 2023-11-15T06:45:17.808 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:17.808 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:17.809 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f466c02ff10 msgr2=0x7f466c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:17.809 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f466c02ff10 0x7f466c0323d0 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f46780096a0 tx=0x7f4678006b40).stop 2023-11-15T06:45:17.810 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 msgr2=0x7f46841064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:17.810 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46841064b0 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f4674009830 tx=0x7f467400b040).stop 2023-11-15T06:45:17.810 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 shutdown_connections 2023-11-15T06:45:17.810 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f466c02ff10 0x7f466c0323d0 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:17.810 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 --2- 172.21.15.135:0/615537466 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f46840f62a0 0x7f46841064b0 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:17.811 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 >> 172.21.15.135:0/615537466 conn(0x7f46840f1aa0 msgr2=0x7f46840f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:17.811 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 shutdown_connections 2023-11-15T06:45:17.811 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:17.806+0000 7f4683fff700 1 -- 172.21.15.135:0/615537466 wait complete. 2023-11-15T06:45:17.812 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:17.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:16.302773+0000 mgr.smithi123.razjtn (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:18.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:18 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:17.805960+0000 mon.smithi123 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.135:0/615537466' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:19.188 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:19.189 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:19.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:18.303110+0000 mgr.smithi123.razjtn (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:20.348 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.343+0000 7f686ba22700 1 -- 172.21.15.135:0/888566651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 msgr2=0x7f68640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:20.348 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.343+0000 7f686ba22700 1 --2- 172.21.15.135:0/888566651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f68640f6930 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7f6854009fb0 tx=0x7f6854009b30).stop 2023-11-15T06:45:20.348 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.343+0000 7f686ba22700 1 -- 172.21.15.135:0/888566651 shutdown_connections 2023-11-15T06:45:20.348 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.343+0000 7f686ba22700 1 --2- 172.21.15.135:0/888566651 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f68640f6930 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:20.349 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.343+0000 7f686ba22700 1 -- 172.21.15.135:0/888566651 >> 172.21.15.135:0/888566651 conn(0x7f68640f1c70 msgr2=0x7f68640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:20.349 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.343+0000 7f686ba22700 1 -- 172.21.15.135:0/888566651 shutdown_connections 2023-11-15T06:45:20.349 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.343+0000 7f686ba22700 1 -- 172.21.15.135:0/888566651 wait complete. 2023-11-15T06:45:20.349 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f686ba22700 1 Processor -- start 2023-11-15T06:45:20.350 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f686ba22700 1 -- start start 2023-11-15T06:45:20.350 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f686ba22700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f6864106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:20.350 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f686ba22700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6864106c20 con 0x7f68640f6510 2023-11-15T06:45:20.350 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f68697be700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f6864106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:20.350 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f68697be700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f6864106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:39838/0 (socket says 172.21.15.135:39838) 2023-11-15T06:45:20.350 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f68697be700 1 -- 172.21.15.135:0/186798256 learned_addr learned my addr 172.21.15.135:0/186798256 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:20.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.344+0000 7f68697be700 1 -- 172.21.15.135:0/186798256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6854005040 con 0x7f68640f6510 2023-11-15T06:45:20.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.345+0000 7f68697be700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f6864106730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6854013510 tx=0x7f685400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:20.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.345+0000 7f6859ffb700 1 -- 172.21.15.135:0/186798256 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6854009320 con 0x7f68640f6510 2023-11-15T06:45:20.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.345+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6864106e20 con 0x7f68640f6510 2023-11-15T06:45:20.351 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.345+0000 7f6859ffb700 1 -- 172.21.15.135:0/186798256 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6854013f00 con 0x7f68640f6510 2023-11-15T06:45:20.352 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.345+0000 7f6859ffb700 1 -- 172.21.15.135:0/186798256 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6854014d90 con 0x7f68640f6510 2023-11-15T06:45:20.352 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.345+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68641072b0 con 0x7f68640f6510 2023-11-15T06:45:20.354 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.346+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6864045ad0 con 0x7f68640f6510 2023-11-15T06:45:20.354 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.346+0000 7f6859ffb700 1 -- 172.21.15.135:0/186798256 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68540095f0 con 0x7f68640f6510 2023-11-15T06:45:20.355 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.346+0000 7f6859ffb700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f685002ff60 0x7f6850032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:20.355 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.346+0000 7f6859ffb700 1 -- 172.21.15.135:0/186798256 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6854041080 con 0x7f68640f6510 2023-11-15T06:45:20.355 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.346+0000 7f6868fbd700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f685002ff60 0x7f6850032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:20.355 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.347+0000 7f6868fbd700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f685002ff60 0x7f6850032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f68600096a0 tx=0x7f6860006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:20.355 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.351+0000 7f6859ffb700 1 -- 172.21.15.135:0/186798256 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6854014420 con 0x7f68640f6510 2023-11-15T06:45:20.560 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.558+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68640fbf60 con 0x7f68640f6510 2023-11-15T06:45:20.561 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.558+0000 7f6859ffb700 1 -- 172.21.15.135:0/186798256 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6854010230 con 0x7f68640f6510 2023-11-15T06:45:20.562 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:20.562 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:20.563 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f685002ff60 msgr2=0x7f6850032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:20.563 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f685002ff60 0x7f6850032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7f68600096a0 tx=0x7f6860006b40).stop 2023-11-15T06:45:20.563 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 msgr2=0x7f6864106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:20.563 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f6864106730 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7f6854013510 tx=0x7f685400b040).stop 2023-11-15T06:45:20.564 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 shutdown_connections 2023-11-15T06:45:20.564 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f685002ff60 0x7f6850032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:20.564 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 --2- 172.21.15.135:0/186798256 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f68640f6510 0x7f6864106730 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:20.564 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 >> 172.21.15.135:0/186798256 conn(0x7f68640f1c70 msgr2=0x7f68640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:20.564 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 shutdown_connections 2023-11-15T06:45:20.565 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:20.560+0000 7f686ba22700 1 -- 172.21.15.135:0/186798256 wait complete. 2023-11-15T06:45:20.565 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:20.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:20 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:20.559608+0000 mon.smithi123 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.135:0/186798256' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:21.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:20.303458+0000 mgr.smithi123.razjtn (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:21.961 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:21.962 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:23.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.139+0000 7fb3cda4d700 1 -- 172.21.15.135:0/2814442783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 msgr2=0x7fb3c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:23.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.139+0000 7fb3cda4d700 1 --2- 172.21.15.135:0/2814442783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c80f5830 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7fb3bc009fb0 tx=0x7fb3bc009b30).stop 2023-11-15T06:45:23.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.139+0000 7fb3cda4d700 1 -- 172.21.15.135:0/2814442783 shutdown_connections 2023-11-15T06:45:23.145 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.139+0000 7fb3cda4d700 1 --2- 172.21.15.135:0/2814442783 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c80f5830 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:23.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.139+0000 7fb3cda4d700 1 -- 172.21.15.135:0/2814442783 >> 172.21.15.135:0/2814442783 conn(0x7fb3c80f0b70 msgr2=0x7fb3c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:23.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.139+0000 7fb3cda4d700 1 -- 172.21.15.135:0/2814442783 shutdown_connections 2023-11-15T06:45:23.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.139+0000 7fb3cda4d700 1 -- 172.21.15.135:0/2814442783 wait complete. 2023-11-15T06:45:23.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.140+0000 7fb3cda4d700 1 Processor -- start 2023-11-15T06:45:23.146 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.140+0000 7fb3cda4d700 1 -- start start 2023-11-15T06:45:23.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.140+0000 7fb3cda4d700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:23.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.140+0000 7fb3cda4d700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb3c8105b20 con 0x7fb3c80f5410 2023-11-15T06:45:23.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.140+0000 7fb3c6ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:23.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.140+0000 7fb3c6ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:39844/0 (socket says 172.21.15.135:39844) 2023-11-15T06:45:23.147 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.140+0000 7fb3c6ffd700 1 -- 172.21.15.135:0/3251550427 learned_addr learned my addr 172.21.15.135:0/3251550427 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:23.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.141+0000 7fb3c6ffd700 1 -- 172.21.15.135:0/3251550427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb3bc005040 con 0x7fb3c80f5410 2023-11-15T06:45:23.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.141+0000 7fb3c6ffd700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c8105630 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fb3bc013510 tx=0x7fb3bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:23.148 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.141+0000 7fb3af7fe700 1 -- 172.21.15.135:0/3251550427 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3bc009320 con 0x7fb3c80f5410 2023-11-15T06:45:23.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.141+0000 7fb3af7fe700 1 -- 172.21.15.135:0/3251550427 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb3bc013f00 con 0x7fb3c80f5410 2023-11-15T06:45:23.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.141+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb3c8105d20 con 0x7fb3c80f5410 2023-11-15T06:45:23.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.141+0000 7fb3af7fe700 1 -- 172.21.15.135:0/3251550427 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb3bc014d90 con 0x7fb3c80f5410 2023-11-15T06:45:23.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.141+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb3c81061b0 con 0x7fb3c80f5410 2023-11-15T06:45:23.149 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.142+0000 7fb3af7fe700 1 -- 172.21.15.135:0/3251550427 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb3bc0095f0 con 0x7fb3c80f5410 2023-11-15T06:45:23.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.142+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb3c8045ad0 con 0x7fb3c80f5410 2023-11-15T06:45:23.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.142+0000 7fb3af7fe700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb3b002ffb0 0x7fb3b0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:23.150 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.142+0000 7fb3af7fe700 1 -- 172.21.15.135:0/3251550427 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb3bc0410a0 con 0x7fb3c80f5410 2023-11-15T06:45:23.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.142+0000 7fb3c67fc700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb3b002ffb0 0x7fb3b0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:23.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.143+0000 7fb3c67fc700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb3b002ffb0 0x7fb3b0032470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb3b80096a0 tx=0x7fb3b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:23.151 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.147+0000 7fb3af7fe700 1 -- 172.21.15.135:0/3251550427 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb3bc06d2b0 con 0x7fb3c80f5410 2023-11-15T06:45:23.358 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.355+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb3c80fae60 con 0x7fb3c80f5410 2023-11-15T06:45:23.359 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.356+0000 7fb3af7fe700 1 -- 172.21.15.135:0/3251550427 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb3bc014420 con 0x7fb3c80f5410 2023-11-15T06:45:23.359 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:23.360 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:23.360 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb3b002ffb0 msgr2=0x7fb3b0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:23.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb3b002ffb0 0x7fb3b0032470 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb3b80096a0 tx=0x7fb3b8006b40).stop 2023-11-15T06:45:23.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 msgr2=0x7fb3c8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:23.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c8105630 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7fb3bc013510 tx=0x7fb3bc00b040).stop 2023-11-15T06:45:23.361 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 shutdown_connections 2023-11-15T06:45:23.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fb3b002ffb0 0x7fb3b0032470 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:23.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 --2- 172.21.15.135:0/3251550427 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fb3c80f5410 0x7fb3c8105630 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:23.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.357+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 >> 172.21.15.135:0/3251550427 conn(0x7fb3c80f0b70 msgr2=0x7fb3c80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:23.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.358+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 shutdown_connections 2023-11-15T06:45:23.362 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:23.358+0000 7fb3cda4d700 1 -- 172.21.15.135:0/3251550427 wait complete. 2023-11-15T06:45:23.363 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:23.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:22.303825+0000 mgr.smithi123.razjtn (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:23.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:23.357149+0000 mon.smithi123 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.135:0/3251550427' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:24.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:24.750 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:24.304166+0000 mgr.smithi123.razjtn (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:25.917 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.913+0000 7f3782213700 1 -- 172.21.15.135:0/1998550225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 msgr2=0x7f377c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:25.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.913+0000 7f3782213700 1 --2- 172.21.15.135:0/1998550225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c0f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7f3770009fb0 tx=0x7f3770009b30).stop 2023-11-15T06:45:25.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.913+0000 7f3782213700 1 -- 172.21.15.135:0/1998550225 shutdown_connections 2023-11-15T06:45:25.918 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.913+0000 7f3782213700 1 --2- 172.21.15.135:0/1998550225 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c0f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:25.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.913+0000 7f3782213700 1 -- 172.21.15.135:0/1998550225 >> 172.21.15.135:0/1998550225 conn(0x7f377c0f0b70 msgr2=0x7f377c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:25.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.913+0000 7f3782213700 1 -- 172.21.15.135:0/1998550225 shutdown_connections 2023-11-15T06:45:25.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.913+0000 7f3782213700 1 -- 172.21.15.135:0/1998550225 wait complete. 2023-11-15T06:45:25.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.914+0000 7f3782213700 1 Processor -- start 2023-11-15T06:45:25.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.914+0000 7f3782213700 1 -- start start 2023-11-15T06:45:25.919 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.914+0000 7f3782213700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:25.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.914+0000 7f3782213700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f377c105b20 con 0x7f377c0f5410 2023-11-15T06:45:25.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.914+0000 7f377b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:25.920 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.914+0000 7f377b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:39854/0 (socket says 172.21.15.135:39854) 2023-11-15T06:45:25.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.914+0000 7f377b7fe700 1 -- 172.21.15.135:0/4026632437 learned_addr learned my addr 172.21.15.135:0/4026632437 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:25.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.915+0000 7f377b7fe700 1 -- 172.21.15.135:0/4026632437 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3770005040 con 0x7f377c0f5410 2023-11-15T06:45:25.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.915+0000 7f377b7fe700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c105630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f3770013510 tx=0x7f377000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:25.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.915+0000 7f3763fff700 1 -- 172.21.15.135:0/4026632437 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3770009320 con 0x7f377c0f5410 2023-11-15T06:45:25.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.915+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f377c105d20 con 0x7f377c0f5410 2023-11-15T06:45:25.921 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.915+0000 7f3763fff700 1 -- 172.21.15.135:0/4026632437 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3770013f00 con 0x7f377c0f5410 2023-11-15T06:45:25.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.915+0000 7f3763fff700 1 -- 172.21.15.135:0/4026632437 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3770014d90 con 0x7f377c0f5410 2023-11-15T06:45:25.922 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.915+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f377c1061b0 con 0x7f377c0f5410 2023-11-15T06:45:25.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.916+0000 7f3763fff700 1 -- 172.21.15.135:0/4026632437 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3770014420 con 0x7f377c0f5410 2023-11-15T06:45:25.923 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.916+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f377c045ad0 con 0x7f377c0f5410 2023-11-15T06:45:25.924 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.916+0000 7f3763fff700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f376402ff60 0x7f3764032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:25.924 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.916+0000 7f3763fff700 1 -- 172.21.15.135:0/4026632437 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3770041070 con 0x7f377c0f5410 2023-11-15T06:45:25.924 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.916+0000 7f377affd700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f376402ff60 0x7f3764032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:25.924 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.917+0000 7f377affd700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f376402ff60 0x7f3764032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f376c0096a0 tx=0x7f376c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:25.924 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:25.921+0000 7f3763fff700 1 -- 172.21.15.135:0/4026632437 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37700095f0 con 0x7f377c0f5410 2023-11-15T06:45:26.134 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.130+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f377c0fae60 con 0x7f377c0f5410 2023-11-15T06:45:26.134 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.131+0000 7f3763fff700 1 -- 172.21.15.135:0/4026632437 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f377001db70 con 0x7f377c0f5410 2023-11-15T06:45:26.134 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:26.135 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:26.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f376402ff60 msgr2=0x7f3764032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:26.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f376402ff60 0x7f3764032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7f376c0096a0 tx=0x7f376c006b40).stop 2023-11-15T06:45:26.136 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 msgr2=0x7f377c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:26.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c105630 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7f3770013510 tx=0x7f377000b040).stop 2023-11-15T06:45:26.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 shutdown_connections 2023-11-15T06:45:26.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f376402ff60 0x7f3764032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:26.137 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 --2- 172.21.15.135:0/4026632437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f377c0f5410 0x7f377c105630 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:26.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.132+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 >> 172.21.15.135:0/4026632437 conn(0x7f377c0f0b70 msgr2=0x7f377c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:26.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.133+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 shutdown_connections 2023-11-15T06:45:26.138 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:26.133+0000 7f3782213700 1 -- 172.21.15.135:0/4026632437 wait complete. 2023-11-15T06:45:26.138 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:26.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:26 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:26.132057+0000 mon.smithi123 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.135:0/4026632437' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:27.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:27.526 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:27.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:26.304528+0000 mgr.smithi123.razjtn (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:28.736 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.732+0000 7f763e5d9700 1 -- 172.21.15.135:0/986709313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 msgr2=0x7f76380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:28.736 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.732+0000 7f763e5d9700 1 --2- 172.21.15.135:0/986709313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f76380f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7f7628009fb0 tx=0x7f7628009b30).stop 2023-11-15T06:45:28.736 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.732+0000 7f763e5d9700 1 -- 172.21.15.135:0/986709313 shutdown_connections 2023-11-15T06:45:28.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.732+0000 7f763e5d9700 1 --2- 172.21.15.135:0/986709313 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f76380f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:28.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.732+0000 7f763e5d9700 1 -- 172.21.15.135:0/986709313 >> 172.21.15.135:0/986709313 conn(0x7f76380f0b70 msgr2=0x7f76380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:28.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.733+0000 7f763e5d9700 1 -- 172.21.15.135:0/986709313 shutdown_connections 2023-11-15T06:45:28.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.733+0000 7f763e5d9700 1 -- 172.21.15.135:0/986709313 wait complete. 2023-11-15T06:45:28.737 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.733+0000 7f763e5d9700 1 Processor -- start 2023-11-15T06:45:28.739 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.733+0000 7f763e5d9700 1 -- start start 2023-11-15T06:45:28.739 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f763e5d9700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f7638109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:28.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f763e5d9700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7638060220 con 0x7f76380f5410 2023-11-15T06:45:28.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7637fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f7638109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:28.740 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7637fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f7638109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33356/0 (socket says 172.21.15.135:33356) 2023-11-15T06:45:28.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7637fff700 1 -- 172.21.15.135:0/2304180447 learned_addr learned my addr 172.21.15.135:0/2304180447 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:28.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7637fff700 1 -- 172.21.15.135:0/2304180447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7628005040 con 0x7f76380f5410 2023-11-15T06:45:28.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7637fff700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f7638109450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f7628013510 tx=0x7f762800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:28.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7634ff9700 1 -- 172.21.15.135:0/2304180447 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7628009320 con 0x7f76380f5410 2023-11-15T06:45:28.741 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7638060420 con 0x7f76380f5410 2023-11-15T06:45:28.742 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7634ff9700 1 -- 172.21.15.135:0/2304180447 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7628013f00 con 0x7f76380f5410 2023-11-15T06:45:28.742 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.734+0000 7f7634ff9700 1 -- 172.21.15.135:0/2304180447 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7628014d90 con 0x7f76380f5410 2023-11-15T06:45:28.742 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.735+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7638109b00 con 0x7f76380f5410 2023-11-15T06:45:28.742 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.735+0000 7f7634ff9700 1 -- 172.21.15.135:0/2304180447 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7628014420 con 0x7f76380f5410 2023-11-15T06:45:28.742 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.735+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7638059100 con 0x7f76380f5410 2023-11-15T06:45:28.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.735+0000 7f7634ff9700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f762402ff60 0x7f7624032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:28.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.735+0000 7f7634ff9700 1 -- 172.21.15.135:0/2304180447 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76280410c0 con 0x7f76380f5410 2023-11-15T06:45:28.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.736+0000 7f76377fe700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f762402ff60 0x7f7624032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:28.743 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.736+0000 7f76377fe700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f762402ff60 0x7f7624032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f76200096a0 tx=0x7f7620006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:28.744 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.740+0000 7f7634ff9700 1 -- 172.21.15.135:0/2304180447 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76280095f0 con 0x7f76380f5410 2023-11-15T06:45:28.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.944+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7638045ad0 con 0x7f76380f5410 2023-11-15T06:45:28.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.944+0000 7f7634ff9700 1 -- 172.21.15.135:0/2304180447 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f762801db70 con 0x7f76380f5410 2023-11-15T06:45:28.948 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:28.948 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:28.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f762402ff60 msgr2=0x7f7624032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:28.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f762402ff60 0x7f7624032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f76200096a0 tx=0x7f7620006b40).stop 2023-11-15T06:45:28.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 msgr2=0x7f7638109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:28.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f7638109450 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7f7628013510 tx=0x7f762800b040).stop 2023-11-15T06:45:28.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 shutdown_connections 2023-11-15T06:45:28.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f762402ff60 0x7f7624032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:28.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 --2- 172.21.15.135:0/2304180447 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f76380f5410 0x7f7638109450 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:28.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 >> 172.21.15.135:0/2304180447 conn(0x7f76380f0b70 msgr2=0x7f76380f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:28.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 shutdown_connections 2023-11-15T06:45:28.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:28.946+0000 7f763e5d9700 1 -- 172.21.15.135:0/2304180447 wait complete. 2023-11-15T06:45:28.953 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:30.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:28.304920+0000 mgr.smithi123.razjtn (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:30.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:28.945622+0000 mon.smithi123 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.135:0/2304180447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:30.269 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:30.269 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:31.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.411+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/2164313072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 msgr2=0x7ffbac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:31.416 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.411+0000 7ffbb2cf3700 1 --2- 172.21.15.135:0/2164313072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac0f5830 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7ffba0009fb0 tx=0x7ffba0009b30).stop 2023-11-15T06:45:31.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.411+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/2164313072 shutdown_connections 2023-11-15T06:45:31.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.411+0000 7ffbb2cf3700 1 --2- 172.21.15.135:0/2164313072 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac0f5830 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:31.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.411+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/2164313072 >> 172.21.15.135:0/2164313072 conn(0x7ffbac0f0b70 msgr2=0x7ffbac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:31.417 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.411+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/2164313072 shutdown_connections 2023-11-15T06:45:31.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.411+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/2164313072 wait complete. 2023-11-15T06:45:31.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.412+0000 7ffbb2cf3700 1 Processor -- start 2023-11-15T06:45:31.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.412+0000 7ffbb2cf3700 1 -- start start 2023-11-15T06:45:31.418 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.412+0000 7ffbb2cf3700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:31.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.412+0000 7ffbb2cf3700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffbac060280 con 0x7ffbac0f5410 2023-11-15T06:45:31.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffbb0a8f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:31.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffbb0a8f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33378/0 (socket says 172.21.15.135:33378) 2023-11-15T06:45:31.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffbb0a8f700 1 -- 172.21.15.135:0/1215154091 learned_addr learned my addr 172.21.15.135:0/1215154091 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:31.419 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffbb0a8f700 1 -- 172.21.15.135:0/1215154091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffba0005040 con 0x7ffbac0f5410 2023-11-15T06:45:31.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffbb0a8f700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac1093f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7ffba0013510 tx=0x7ffba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:31.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffba97fa700 1 -- 172.21.15.135:0/1215154091 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba0009320 con 0x7ffbac0f5410 2023-11-15T06:45:31.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffbac060480 con 0x7ffbac0f5410 2023-11-15T06:45:31.421 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffba97fa700 1 -- 172.21.15.135:0/1215154091 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffba0013f00 con 0x7ffbac0f5410 2023-11-15T06:45:31.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffba97fa700 1 -- 172.21.15.135:0/1215154091 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffba0014d90 con 0x7ffbac0f5410 2023-11-15T06:45:31.423 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.413+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffbac109b20 con 0x7ffbac0f5410 2023-11-15T06:45:31.424 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.414+0000 7ffba97fa700 1 -- 172.21.15.135:0/1215154091 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffba00263a0 con 0x7ffbac0f5410 2023-11-15T06:45:31.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.414+0000 7ffba97fa700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb9c02ff60 0x7ffb9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:31.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.414+0000 7ffba97fa700 1 -- 172.21.15.135:0/1215154091 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffba0041010 con 0x7ffbac0f5410 2023-11-15T06:45:31.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.414+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffbac045ad0 con 0x7ffbac0f5410 2023-11-15T06:45:31.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.414+0000 7ffbabfff700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb9c02ff60 0x7ffb9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:31.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.415+0000 7ffbabfff700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb9c02ff60 0x7ffb9c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ffb980096a0 tx=0x7ffb98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:31.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.419+0000 7ffba97fa700 1 -- 172.21.15.135:0/1215154091 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffba0012920 con 0x7ffbac0f5410 2023-11-15T06:45:31.628 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.625+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffbac10a240 con 0x7ffbac0f5410 2023-11-15T06:45:31.629 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.626+0000 7ffba97fa700 1 -- 172.21.15.135:0/1215154091 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffba002b020 con 0x7ffbac0f5410 2023-11-15T06:45:31.629 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:31.630 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:31.631 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb9c02ff60 msgr2=0x7ffb9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:31.631 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb9c02ff60 0x7ffb9c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7ffb980096a0 tx=0x7ffb98006b40).stop 2023-11-15T06:45:31.631 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 msgr2=0x7ffbac1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:31.631 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac1093f0 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7ffba0013510 tx=0x7ffba000b040).stop 2023-11-15T06:45:31.631 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 shutdown_connections 2023-11-15T06:45:31.632 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb9c02ff60 0x7ffb9c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:31.632 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 --2- 172.21.15.135:0/1215154091 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffbac0f5410 0x7ffbac1093f0 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:31.632 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 >> 172.21.15.135:0/1215154091 conn(0x7ffbac0f0b70 msgr2=0x7ffbac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:31.632 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 shutdown_connections 2023-11-15T06:45:31.633 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:31.628+0000 7ffbb2cf3700 1 -- 172.21.15.135:0/1215154091 wait complete. 2023-11-15T06:45:31.633 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:32.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:30.305261+0000 mgr.smithi123.razjtn (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:32.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:31.627208+0000 mon.smithi123 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.135:0/1215154091' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:33.017 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:33.018 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:34.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:32.305640+0000 mgr.smithi123.razjtn (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:34.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 -- 172.21.15.135:0/794117221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 msgr2=0x7fe64c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:34.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 --2- 172.21.15.135:0/794117221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f90f0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7fe640009fb0 tx=0x7fe640009b30).stop 2023-11-15T06:45:34.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 -- 172.21.15.135:0/794117221 shutdown_connections 2023-11-15T06:45:34.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 --2- 172.21.15.135:0/794117221 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f90f0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:34.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 -- 172.21.15.135:0/794117221 >> 172.21.15.135:0/794117221 conn(0x7fe64c0f0b70 msgr2=0x7fe64c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:34.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 -- 172.21.15.135:0/794117221 shutdown_connections 2023-11-15T06:45:34.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 -- 172.21.15.135:0/794117221 wait complete. 2023-11-15T06:45:34.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 Processor -- start 2023-11-15T06:45:34.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.177+0000 7fe6513f6700 1 -- start start 2023-11-15T06:45:34.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe6513f6700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:34.183 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe6513f6700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe64c0f4ff0 con 0x7fe64c060280 2023-11-15T06:45:34.183 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe64affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:34.183 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe64affd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33394/0 (socket says 172.21.15.135:33394) 2023-11-15T06:45:34.183 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe64affd700 1 -- 172.21.15.135:0/264395538 learned_addr learned my addr 172.21.15.135:0/264395538 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:34.184 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe64affd700 1 -- 172.21.15.135:0/264395538 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe640005040 con 0x7fe64c060280 2023-11-15T06:45:34.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe64affd700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f6b70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fe640009830 tx=0x7fe64000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:34.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe6337fe700 1 -- 172.21.15.135:0/264395538 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe640009320 con 0x7fe64c060280 2023-11-15T06:45:34.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe64c0f5190 con 0x7fe64c060280 2023-11-15T06:45:34.185 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe6337fe700 1 -- 172.21.15.135:0/264395538 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe6400136c0 con 0x7fe64c060280 2023-11-15T06:45:34.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.178+0000 7fe6337fe700 1 -- 172.21.15.135:0/264395538 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe64001ee90 con 0x7fe64c060280 2023-11-15T06:45:34.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.179+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe64c0f5620 con 0x7fe64c060280 2023-11-15T06:45:34.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.179+0000 7fe6337fe700 1 -- 172.21.15.135:0/264395538 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe64001e420 con 0x7fe64c060280 2023-11-15T06:45:34.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.179+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe64c059100 con 0x7fe64c060280 2023-11-15T06:45:34.186 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.179+0000 7fe6337fe700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe63402ff10 0x7fe6340323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:34.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.179+0000 7fe6337fe700 1 -- 172.21.15.135:0/264395538 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe640040f10 con 0x7fe64c060280 2023-11-15T06:45:34.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.180+0000 7fe64a7fc700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe63402ff10 0x7fe6340323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:34.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.180+0000 7fe64a7fc700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe63402ff10 0x7fe6340323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe63c0096a0 tx=0x7fe63c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:34.187 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.184+0000 7fe6337fe700 1 -- 172.21.15.135:0/264395538 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe64001c790 con 0x7fe64c060280 2023-11-15T06:45:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.394+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe64c0fbf50 con 0x7fe64c060280 2023-11-15T06:45:34.397 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.394+0000 7fe6337fe700 1 -- 172.21.15.135:0/264395538 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe64002c2d0 con 0x7fe64c060280 2023-11-15T06:45:34.398 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:34.398 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:34.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe63402ff10 msgr2=0x7fe6340323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:34.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe63402ff10 0x7fe6340323d0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7fe63c0096a0 tx=0x7fe63c006b40).stop 2023-11-15T06:45:34.399 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 msgr2=0x7fe64c0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:34.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f6b70 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7fe640009830 tx=0x7fe64000b040).stop 2023-11-15T06:45:34.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 shutdown_connections 2023-11-15T06:45:34.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fe63402ff10 0x7fe6340323d0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:34.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 --2- 172.21.15.135:0/264395538 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fe64c060280 0x7fe64c0f6b70 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:34.400 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 >> 172.21.15.135:0/264395538 conn(0x7fe64c0f0b70 msgr2=0x7fe64c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 shutdown_connections 2023-11-15T06:45:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:34.396+0000 7fe6513f6700 1 -- 172.21.15.135:0/264395538 wait complete. 2023-11-15T06:45:34.401 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:35.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:34 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:34.395554+0000 mon.smithi123 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.135:0/264395538' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:35.816 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:35.816 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:36.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:34.305989+0000 mgr.smithi123.razjtn (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:36.976 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.972+0000 7fd01baf5700 1 -- 172.21.15.135:0/4196273459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 msgr2=0x7fd0140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:36.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.972+0000 7fd01baf5700 1 --2- 172.21.15.135:0/4196273459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd0140f5830 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7fd004009fb0 tx=0x7fd004009b30).stop 2023-11-15T06:45:36.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.972+0000 7fd01baf5700 1 -- 172.21.15.135:0/4196273459 shutdown_connections 2023-11-15T06:45:36.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.972+0000 7fd01baf5700 1 --2- 172.21.15.135:0/4196273459 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd0140f5830 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:36.977 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.972+0000 7fd01baf5700 1 -- 172.21.15.135:0/4196273459 >> 172.21.15.135:0/4196273459 conn(0x7fd0140f0b70 msgr2=0x7fd0140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:36.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.972+0000 7fd01baf5700 1 -- 172.21.15.135:0/4196273459 shutdown_connections 2023-11-15T06:45:36.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.972+0000 7fd01baf5700 1 -- 172.21.15.135:0/4196273459 wait complete. 2023-11-15T06:45:36.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd01baf5700 1 Processor -- start 2023-11-15T06:45:36.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd01baf5700 1 -- start start 2023-11-15T06:45:36.978 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd01baf5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd014106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:36.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd01baf5700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd014106c20 con 0x7fd0140f5410 2023-11-15T06:45:36.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd019891700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd014106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:36.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd019891700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd014106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:33410/0 (socket says 172.21.15.135:33410) 2023-11-15T06:45:36.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd019891700 1 -- 172.21.15.135:0/1443284513 learned_addr learned my addr 172.21.15.135:0/1443284513 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:36.979 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.973+0000 7fd019891700 1 -- 172.21.15.135:0/1443284513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd004005040 con 0x7fd0140f5410 2023-11-15T06:45:36.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.974+0000 7fd019891700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd014106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd004013510 tx=0x7fd00400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:36.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.974+0000 7fd00a7fc700 1 -- 172.21.15.135:0/1443284513 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd004009320 con 0x7fd0140f5410 2023-11-15T06:45:36.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.974+0000 7fd00a7fc700 1 -- 172.21.15.135:0/1443284513 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd004013f00 con 0x7fd0140f5410 2023-11-15T06:45:36.980 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.974+0000 7fd00a7fc700 1 -- 172.21.15.135:0/1443284513 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd004014d90 con 0x7fd0140f5410 2023-11-15T06:45:36.981 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.974+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd014106e20 con 0x7fd0140f5410 2023-11-15T06:45:36.981 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.974+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd0141072b0 con 0x7fd0140f5410 2023-11-15T06:45:36.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.975+0000 7fd00a7fc700 1 -- 172.21.15.135:0/1443284513 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0040095f0 con 0x7fd0140f5410 2023-11-15T06:45:36.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.975+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd014045ad0 con 0x7fd0140f5410 2023-11-15T06:45:36.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.975+0000 7fd00a7fc700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd00002ff60 0x7fd000032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:36.982 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.975+0000 7fd00a7fc700 1 -- 172.21.15.135:0/1443284513 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd0040410c0 con 0x7fd0140f5410 2023-11-15T06:45:36.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.975+0000 7fd019090700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd00002ff60 0x7fd000032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:36.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.975+0000 7fd019090700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd00002ff60 0x7fd000032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd0100096a0 tx=0x7fd010006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:36.983 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:36.980+0000 7fd00a7fc700 1 -- 172.21.15.135:0/1443284513 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd00401d7f0 con 0x7fd0140f5410 2023-11-15T06:45:37.192 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.187+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd0140fae60 con 0x7fd0140f5410 2023-11-15T06:45:37.192 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.188+0000 7fd00a7fc700 1 -- 172.21.15.135:0/1443284513 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd004010230 con 0x7fd0140f5410 2023-11-15T06:45:37.193 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:37.193 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:37.194 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.189+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd00002ff60 msgr2=0x7fd000032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:37.194 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.189+0000 7fd01baf5700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd00002ff60 0x7fd000032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7fd0100096a0 tx=0x7fd010006b40).stop 2023-11-15T06:45:37.195 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.189+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 msgr2=0x7fd014106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:37.195 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.189+0000 7fd01baf5700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd014106730 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7fd004013510 tx=0x7fd00400b040).stop 2023-11-15T06:45:37.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.190+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 shutdown_connections 2023-11-15T06:45:37.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.190+0000 7fd01baf5700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd00002ff60 0x7fd000032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:37.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.190+0000 7fd01baf5700 1 --2- 172.21.15.135:0/1443284513 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd0140f5410 0x7fd014106730 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:37.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.190+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 >> 172.21.15.135:0/1443284513 conn(0x7fd0140f0b70 msgr2=0x7fd0140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:37.196 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.190+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 shutdown_connections 2023-11-15T06:45:37.197 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:37.190+0000 7fd01baf5700 1 -- 172.21.15.135:0/1443284513 wait complete. 2023-11-15T06:45:37.197 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:38.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:36.306346+0000 mgr.smithi123.razjtn (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:38.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:37 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:37.189179+0000 mon.smithi123 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.135:0/1443284513' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:38.593 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:38.594 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:39.714 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 -- 172.21.15.135:0/1146667069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 msgr2=0x7f6f700f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:39.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 --2- 172.21.15.135:0/1146667069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f700f6930 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f6f64009fb0 tx=0x7f6f64009b30).stop 2023-11-15T06:45:39.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 -- 172.21.15.135:0/1146667069 shutdown_connections 2023-11-15T06:45:39.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 --2- 172.21.15.135:0/1146667069 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f700f6930 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:39.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 -- 172.21.15.135:0/1146667069 >> 172.21.15.135:0/1146667069 conn(0x7f6f700f1c70 msgr2=0x7f6f700f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:39.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 -- 172.21.15.135:0/1146667069 shutdown_connections 2023-11-15T06:45:39.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 -- 172.21.15.135:0/1146667069 wait complete. 2023-11-15T06:45:39.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.711+0000 7f6f75a68700 1 Processor -- start 2023-11-15T06:45:39.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f75a68700 1 -- start start 2023-11-15T06:45:39.718 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f75a68700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f70106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:39.718 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f75a68700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6f70106c20 con 0x7f6f700f6510 2023-11-15T06:45:39.718 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f6f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f70106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:39.719 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f6f7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f70106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:46874/0 (socket says 172.21.15.135:46874) 2023-11-15T06:45:39.719 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f6f7fe700 1 -- 172.21.15.135:0/3416927097 learned_addr learned my addr 172.21.15.135:0/3416927097 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:39.719 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f6f7fe700 1 -- 172.21.15.135:0/3416927097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f64005040 con 0x7f6f700f6510 2023-11-15T06:45:39.719 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.712+0000 7f6f6f7fe700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f70106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f6f64013510 tx=0x7f6f6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:39.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.713+0000 7f6f57fff700 1 -- 172.21.15.135:0/3416927097 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f64009320 con 0x7f6f700f6510 2023-11-15T06:45:39.722 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.713+0000 7f6f57fff700 1 -- 172.21.15.135:0/3416927097 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f64013f00 con 0x7f6f700f6510 2023-11-15T06:45:39.722 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.713+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6f70106e20 con 0x7f6f700f6510 2023-11-15T06:45:39.722 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.713+0000 7f6f57fff700 1 -- 172.21.15.135:0/3416927097 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f64014d90 con 0x7f6f700f6510 2023-11-15T06:45:39.722 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.713+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6f701072b0 con 0x7f6f700f6510 2023-11-15T06:45:39.723 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.713+0000 7f6f57fff700 1 -- 172.21.15.135:0/3416927097 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f640263a0 con 0x7f6f700f6510 2023-11-15T06:45:39.723 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.714+0000 7f6f57fff700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6f5802ff60 0x7f6f58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:39.723 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.714+0000 7f6f57fff700 1 -- 172.21.15.135:0/3416927097 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f64041f00 con 0x7f6f700f6510 2023-11-15T06:45:39.723 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.714+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6f70045ad0 con 0x7f6f700f6510 2023-11-15T06:45:39.723 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.714+0000 7f6f6effd700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6f5802ff60 0x7f6f58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:39.724 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.714+0000 7f6f6effd700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6f5802ff60 0x7f6f58032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6f600096a0 tx=0x7f6f60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:39.724 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.718+0000 7f6f57fff700 1 -- 172.21.15.135:0/3416927097 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f64012920 con 0x7f6f700f6510 2023-11-15T06:45:39.928 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.925+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6f70107ff0 con 0x7f6f700f6510 2023-11-15T06:45:39.929 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.926+0000 7f6f57fff700 1 -- 172.21.15.135:0/3416927097 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6f6402b020 con 0x7f6f700f6510 2023-11-15T06:45:39.929 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:39.929 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:39.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6f5802ff60 msgr2=0x7f6f58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:39.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6f5802ff60 0x7f6f58032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f6f600096a0 tx=0x7f6f60006b40).stop 2023-11-15T06:45:39.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 msgr2=0x7f6f70106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:39.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f70106730 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f6f64013510 tx=0x7f6f6400b040).stop 2023-11-15T06:45:39.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 shutdown_connections 2023-11-15T06:45:39.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f6f5802ff60 0x7f6f58032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:39.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 --2- 172.21.15.135:0/3416927097 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f6f700f6510 0x7f6f70106730 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:39.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 >> 172.21.15.135:0/3416927097 conn(0x7f6f700f1c70 msgr2=0x7f6f700f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:39.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 shutdown_connections 2023-11-15T06:45:39.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:39.928+0000 7f6f75a68700 1 -- 172.21.15.135:0/3416927097 wait complete. 2023-11-15T06:45:39.933 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:40.156 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:39 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:38.306734+0000 mgr.smithi123.razjtn (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:41.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:39.927059+0000 mon.smithi123 (mon.0) 303 : audit [DBG] from='client.? 172.21.15.135:0/3416927097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:41.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:40.370922+0000 mon.smithi123 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/mirror_snapshot_schedule"}]: dispatch 2023-11-15T06:45:41.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:40 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:40.376304+0000 mon.smithi123 (mon.0) 305 : audit [INF] from='mgr.14162 172.21.15.123:0/923653028' entity='mgr.smithi123.razjtn' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi123.razjtn/trash_purge_schedule"}]: dispatch 2023-11-15T06:45:41.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:41.322 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:42.157 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:41 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:40.307112+0000 mgr.smithi123.razjtn (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:42.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.453+0000 7f7623b5e700 1 -- 172.21.15.135:0/2214101149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 msgr2=0x7f761c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:42.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.453+0000 7f7623b5e700 1 --2- 172.21.15.135:0/2214101149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c0f6930 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7f760c009fb0 tx=0x7f760c009b30).stop 2023-11-15T06:45:42.458 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.453+0000 7f7623b5e700 1 -- 172.21.15.135:0/2214101149 shutdown_connections 2023-11-15T06:45:42.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.453+0000 7f7623b5e700 1 --2- 172.21.15.135:0/2214101149 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c0f6930 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:42.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.453+0000 7f7623b5e700 1 -- 172.21.15.135:0/2214101149 >> 172.21.15.135:0/2214101149 conn(0x7f761c0f1c70 msgr2=0x7f761c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:42.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.453+0000 7f7623b5e700 1 -- 172.21.15.135:0/2214101149 shutdown_connections 2023-11-15T06:45:42.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.453+0000 7f7623b5e700 1 -- 172.21.15.135:0/2214101149 wait complete. 2023-11-15T06:45:42.459 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f7623b5e700 1 Processor -- start 2023-11-15T06:45:42.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f7623b5e700 1 -- start start 2023-11-15T06:45:42.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f7623b5e700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:42.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f7623b5e700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f761c106c20 con 0x7f761c0f6510 2023-11-15T06:45:42.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f76218fa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:42.460 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f76218fa700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:46884/0 (socket says 172.21.15.135:46884) 2023-11-15T06:45:42.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f76218fa700 1 -- 172.21.15.135:0/3187002662 learned_addr learned my addr 172.21.15.135:0/3187002662 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:42.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.454+0000 7f76218fa700 1 -- 172.21.15.135:0/3187002662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f760c005040 con 0x7f761c0f6510 2023-11-15T06:45:42.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.455+0000 7f76218fa700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f760c013510 tx=0x7f760c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:42.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.455+0000 7f76127fc700 1 -- 172.21.15.135:0/3187002662 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f760c009320 con 0x7f761c0f6510 2023-11-15T06:45:42.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.455+0000 7f76127fc700 1 -- 172.21.15.135:0/3187002662 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f760c013f00 con 0x7f761c0f6510 2023-11-15T06:45:42.461 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.455+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f761c106e20 con 0x7f761c0f6510 2023-11-15T06:45:42.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.455+0000 7f76127fc700 1 -- 172.21.15.135:0/3187002662 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f760c014d90 con 0x7f761c0f6510 2023-11-15T06:45:42.462 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.455+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f761c1072b0 con 0x7f761c0f6510 2023-11-15T06:45:42.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.456+0000 7f76127fc700 1 -- 172.21.15.135:0/3187002662 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f760c0095f0 con 0x7f761c0f6510 2023-11-15T06:45:42.463 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.456+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f761c045ad0 con 0x7f761c0f6510 2023-11-15T06:45:42.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.456+0000 7f76127fc700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f760802ff60 0x7f7608032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:42.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.456+0000 7f76127fc700 1 -- 172.21.15.135:0/3187002662 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f760c040fe0 con 0x7f761c0f6510 2023-11-15T06:45:42.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.456+0000 7f76210f9700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f760802ff60 0x7f7608032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:42.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.456+0000 7f76210f9700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f760802ff60 0x7f7608032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f76180096a0 tx=0x7f7618006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:42.464 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.460+0000 7f76127fc700 1 -- 172.21.15.135:0/3187002662 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f760c01d7f0 con 0x7f761c0f6510 2023-11-15T06:45:42.667 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.664+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f761c0fbf60 con 0x7f761c0f6510 2023-11-15T06:45:42.668 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.665+0000 7f76127fc700 1 -- 172.21.15.135:0/3187002662 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f760c010230 con 0x7f761c0f6510 2023-11-15T06:45:42.668 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:42.669 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:42.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f760802ff60 msgr2=0x7f7608032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:42.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f760802ff60 0x7f7608032420 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7f76180096a0 tx=0x7f7618006b40).stop 2023-11-15T06:45:42.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 msgr2=0x7f761c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:42.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c106730 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7f760c013510 tx=0x7f760c00b040).stop 2023-11-15T06:45:42.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 shutdown_connections 2023-11-15T06:45:42.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f760802ff60 0x7f7608032420 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:42.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 --2- 172.21.15.135:0/3187002662 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f761c0f6510 0x7f761c106730 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:42.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 >> 172.21.15.135:0/3187002662 conn(0x7f761c0f1c70 msgr2=0x7f761c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:42.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 shutdown_connections 2023-11-15T06:45:42.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:42.667+0000 7f7623b5e700 1 -- 172.21.15.135:0/3187002662 wait complete. 2023-11-15T06:45:42.672 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:43.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:42.307473+0000 mgr.smithi123.razjtn (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:43.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:43 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:42.666268+0000 mon.smithi123 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.135:0/3187002662' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:44.045 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:44.046 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:45.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.221+0000 7f43c164a700 1 -- 172.21.15.135:0/3220722373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 msgr2=0x7f43bc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:45.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.221+0000 7f43c164a700 1 --2- 172.21.15.135:0/3220722373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc0f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f43b0009fb0 tx=0x7f43b0009b30).stop 2023-11-15T06:45:45.226 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.221+0000 7f43c164a700 1 -- 172.21.15.135:0/3220722373 shutdown_connections 2023-11-15T06:45:45.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.221+0000 7f43c164a700 1 --2- 172.21.15.135:0/3220722373 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc0f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:45.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.221+0000 7f43c164a700 1 -- 172.21.15.135:0/3220722373 >> 172.21.15.135:0/3220722373 conn(0x7f43bc0f0b70 msgr2=0x7f43bc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:45.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.221+0000 7f43c164a700 1 -- 172.21.15.135:0/3220722373 shutdown_connections 2023-11-15T06:45:45.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.222+0000 7f43c164a700 1 -- 172.21.15.135:0/3220722373 wait complete. 2023-11-15T06:45:45.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.222+0000 7f43c164a700 1 Processor -- start 2023-11-15T06:45:45.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.222+0000 7f43c164a700 1 -- start start 2023-11-15T06:45:45.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.222+0000 7f43c164a700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:45.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.222+0000 7f43c164a700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f43bc105af0 con 0x7f43bc0f5410 2023-11-15T06:45:45.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43baffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:45.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43baffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:46896/0 (socket says 172.21.15.135:46896) 2023-11-15T06:45:45.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43baffd700 1 -- 172.21.15.135:0/3903663687 learned_addr learned my addr 172.21.15.135:0/3903663687 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:45.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43baffd700 1 -- 172.21.15.135:0/3903663687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43b0005040 con 0x7f43bc0f5410 2023-11-15T06:45:45.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43baffd700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc105600 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f43b0013510 tx=0x7f43b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:45.229 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43a37fe700 1 -- 172.21.15.135:0/3903663687 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b0009320 con 0x7f43bc0f5410 2023-11-15T06:45:45.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f43bc105cf0 con 0x7f43bc0f5410 2023-11-15T06:45:45.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43a37fe700 1 -- 172.21.15.135:0/3903663687 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43b0013f00 con 0x7f43bc0f5410 2023-11-15T06:45:45.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43a37fe700 1 -- 172.21.15.135:0/3903663687 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43b0014d90 con 0x7f43bc0f5410 2023-11-15T06:45:45.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.223+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f43bc106180 con 0x7f43bc0f5410 2023-11-15T06:45:45.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.224+0000 7f43a37fe700 1 -- 172.21.15.135:0/3903663687 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43b0014420 con 0x7f43bc0f5410 2023-11-15T06:45:45.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.224+0000 7f43a37fe700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43a402ff60 0x7f43a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:45.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.224+0000 7f43a37fe700 1 -- 172.21.15.135:0/3903663687 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43b0041070 con 0x7f43bc0f5410 2023-11-15T06:45:45.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.224+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f43bc045ad0 con 0x7f43bc0f5410 2023-11-15T06:45:45.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.224+0000 7f43ba7fc700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43a402ff60 0x7f43a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:45.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.225+0000 7f43ba7fc700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43a402ff60 0x7f43a4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f43ac0096a0 tx=0x7f43ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:45.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.229+0000 7f43a37fe700 1 -- 172.21.15.135:0/3903663687 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43b00095f0 con 0x7f43bc0f5410 2023-11-15T06:45:45.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.434+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f43bc106ea0 con 0x7f43bc0f5410 2023-11-15T06:45:45.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.434+0000 7f43a37fe700 1 -- 172.21.15.135:0/3903663687 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f43b001db70 con 0x7f43bc0f5410 2023-11-15T06:45:45.438 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:45.438 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:45.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43a402ff60 msgr2=0x7f43a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:45.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43a402ff60 0x7f43a4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f43ac0096a0 tx=0x7f43ac006b40).stop 2023-11-15T06:45:45.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 msgr2=0x7f43bc105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:45.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc105600 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f43b0013510 tx=0x7f43b000b040).stop 2023-11-15T06:45:45.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 shutdown_connections 2023-11-15T06:45:45.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43a402ff60 0x7f43a4032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:45.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 --2- 172.21.15.135:0/3903663687 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f43bc0f5410 0x7f43bc105600 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:45.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 >> 172.21.15.135:0/3903663687 conn(0x7f43bc0f0b70 msgr2=0x7f43bc0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:45.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 shutdown_connections 2023-11-15T06:45:45.441 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:45.436+0000 7f43c164a700 1 -- 172.21.15.135:0/3903663687 wait complete. 2023-11-15T06:45:45.441 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:45.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:44.307813+0000 mgr.smithi123.razjtn (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:45.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:45 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:45.435558+0000 mon.smithi123 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.135:0/3903663687' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:46.814 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:46.815 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:47.909 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:47 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:46.308143+0000 mgr.smithi123.razjtn (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:47.964 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.959+0000 7fa6882e0700 1 -- 172.21.15.135:0/3100307003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 msgr2=0x7fa6800f90b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:47.964 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.959+0000 7fa6882e0700 1 --2- 172.21.15.135:0/3100307003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa6800f90b0 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7fa670009fb0 tx=0x7fa670009b30).stop 2023-11-15T06:45:47.964 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.959+0000 7fa6882e0700 1 -- 172.21.15.135:0/3100307003 shutdown_connections 2023-11-15T06:45:47.964 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.959+0000 7fa6882e0700 1 --2- 172.21.15.135:0/3100307003 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa6800f90b0 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:47.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.959+0000 7fa6882e0700 1 -- 172.21.15.135:0/3100307003 >> 172.21.15.135:0/3100307003 conn(0x7fa6800f0a90 msgr2=0x7fa6800f2eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:47.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.959+0000 7fa6882e0700 1 -- 172.21.15.135:0/3100307003 shutdown_connections 2023-11-15T06:45:47.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.959+0000 7fa6882e0700 1 -- 172.21.15.135:0/3100307003 wait complete. 2023-11-15T06:45:47.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa6882e0700 1 Processor -- start 2023-11-15T06:45:47.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa6882e0700 1 -- start start 2023-11-15T06:45:47.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa6882e0700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa680106940 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:47.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa6882e0700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa680106e30 con 0x7fa6800601e0 2023-11-15T06:45:47.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa68607c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa680106940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:47.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa68607c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa680106940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:46920/0 (socket says 172.21.15.135:46920) 2023-11-15T06:45:47.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa68607c700 1 -- 172.21.15.135:0/3374240153 learned_addr learned my addr 172.21.15.135:0/3374240153 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:47.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa68607c700 1 -- 172.21.15.135:0/3374240153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa670005040 con 0x7fa6800601e0 2023-11-15T06:45:47.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.960+0000 7fa68607c700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa680106940 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fa6700079e0 tx=0x7fa670009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:47.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.961+0000 7fa676ffd700 1 -- 172.21.15.135:0/3374240153 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa670018070 con 0x7fa6800601e0 2023-11-15T06:45:47.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.961+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa680107030 con 0x7fa6800601e0 2023-11-15T06:45:47.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.961+0000 7fa676ffd700 1 -- 172.21.15.135:0/3374240153 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa670013410 con 0x7fa6800601e0 2023-11-15T06:45:47.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.961+0000 7fa676ffd700 1 -- 172.21.15.135:0/3374240153 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa67001c3c0 con 0x7fa6800601e0 2023-11-15T06:45:47.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.961+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa6801074c0 con 0x7fa6800601e0 2023-11-15T06:45:47.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.961+0000 7fa676ffd700 1 -- 172.21.15.135:0/3374240153 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa67001c630 con 0x7fa6800601e0 2023-11-15T06:45:47.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.961+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa680045ad0 con 0x7fa6800601e0 2023-11-15T06:45:47.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.962+0000 7fa676ffd700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa66c02ff60 0x7fa66c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:47.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.962+0000 7fa676ffd700 1 -- 172.21.15.135:0/3374240153 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa670041790 con 0x7fa6800601e0 2023-11-15T06:45:47.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.962+0000 7fa68587b700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa66c02ff60 0x7fa66c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:47.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.962+0000 7fa68587b700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa66c02ff60 0x7fa66c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa67c0096a0 tx=0x7fa67c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:47.970 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:47.966+0000 7fa676ffd700 1 -- 172.21.15.135:0/3374240153 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa67001ca30 con 0x7fa6800601e0 2023-11-15T06:45:48.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.173+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa68005ea50 con 0x7fa6800601e0 2023-11-15T06:45:48.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.174+0000 7fa676ffd700 1 -- 172.21.15.135:0/3374240153 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa67001b370 con 0x7fa6800601e0 2023-11-15T06:45:48.177 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:48.177 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:48.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa66c02ff60 msgr2=0x7fa66c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:48.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa66c02ff60 0x7fa66c032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7fa67c0096a0 tx=0x7fa67c006b40).stop 2023-11-15T06:45:48.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 msgr2=0x7fa680106940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:48.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa680106940 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7fa6700079e0 tx=0x7fa670009b30).stop 2023-11-15T06:45:48.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 shutdown_connections 2023-11-15T06:45:48.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa66c02ff60 0x7fa66c032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:48.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 --2- 172.21.15.135:0/3374240153 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa6800601e0 0x7fa680106940 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:48.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 >> 172.21.15.135:0/3374240153 conn(0x7fa6800f0a90 msgr2=0x7fa6800f1570 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:48.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 shutdown_connections 2023-11-15T06:45:48.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:48.175+0000 7fa6882e0700 1 -- 172.21.15.135:0/3374240153 wait complete. 2023-11-15T06:45:48.181 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:48.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:48 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:48.174731+0000 mon.smithi123 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.135:0/3374240153' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:49.559 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:49.559 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:49.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:49 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:48.308535+0000 mgr.smithi123.razjtn (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:50.750 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.746+0000 7f4067fff700 1 -- 172.21.15.135:0/3327960175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 msgr2=0x7f40680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:50.750 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.746+0000 7f4067fff700 1 --2- 172.21.15.135:0/3327960175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40680f55c0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f4058009fb0 tx=0x7f4058009b30).stop 2023-11-15T06:45:50.750 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 -- 172.21.15.135:0/3327960175 shutdown_connections 2023-11-15T06:45:50.750 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 --2- 172.21.15.135:0/3327960175 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40680f55c0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:50.751 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 -- 172.21.15.135:0/3327960175 >> 172.21.15.135:0/3327960175 conn(0x7f40680f09a0 msgr2=0x7f40680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:50.751 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 -- 172.21.15.135:0/3327960175 shutdown_connections 2023-11-15T06:45:50.751 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 -- 172.21.15.135:0/3327960175 wait complete. 2023-11-15T06:45:50.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 Processor -- start 2023-11-15T06:45:50.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 -- start start 2023-11-15T06:45:50.753 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40681064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:50.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.747+0000 7f4067fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40681069d0 con 0x7f40680f51a0 2023-11-15T06:45:50.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f4066ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40681064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:50.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f4066ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40681064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:60020/0 (socket says 172.21.15.135:60020) 2023-11-15T06:45:50.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f4066ffd700 1 -- 172.21.15.135:0/4129822727 learned_addr learned my addr 172.21.15.135:0/4129822727 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:50.754 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f4066ffd700 1 -- 172.21.15.135:0/4129822727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4058005040 con 0x7f40680f51a0 2023-11-15T06:45:50.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f4066ffd700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40681064e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4058013510 tx=0x7f405800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:50.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f404f7fe700 1 -- 172.21.15.135:0/4129822727 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4058009320 con 0x7f40680f51a0 2023-11-15T06:45:50.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4068106bd0 con 0x7f40680f51a0 2023-11-15T06:45:50.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f404f7fe700 1 -- 172.21.15.135:0/4129822727 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4058013f00 con 0x7f40680f51a0 2023-11-15T06:45:50.755 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f404f7fe700 1 -- 172.21.15.135:0/4129822727 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4058014dc0 con 0x7f40680f51a0 2023-11-15T06:45:50.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.748+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4068107060 con 0x7f40680f51a0 2023-11-15T06:45:50.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.749+0000 7f404f7fe700 1 -- 172.21.15.135:0/4129822727 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4058014420 con 0x7f40680f51a0 2023-11-15T06:45:50.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.749+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4068045ad0 con 0x7f40680f51a0 2023-11-15T06:45:50.756 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.749+0000 7f404f7fe700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f405002ff60 0x7f4050032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:50.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.749+0000 7f404f7fe700 1 -- 172.21.15.135:0/4129822727 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f40580410f0 con 0x7f40680f51a0 2023-11-15T06:45:50.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.749+0000 7f40667fc700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f405002ff60 0x7f4050032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:50.757 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.750+0000 7f40667fc700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f405002ff60 0x7f4050032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f405c0096a0 tx=0x7f405c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:50.758 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.754+0000 7f404f7fe700 1 -- 172.21.15.135:0/4129822727 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40580095f0 con 0x7f40680f51a0 2023-11-15T06:45:50.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.960+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4068107f10 con 0x7f40680f51a0 2023-11-15T06:45:50.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.960+0000 7f404f7fe700 1 -- 172.21.15.135:0/4129822727 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f405801cb10 con 0x7f40680f51a0 2023-11-15T06:45:50.964 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:50.964 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:50.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f405002ff60 msgr2=0x7f4050032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:50.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f405002ff60 0x7f4050032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f405c0096a0 tx=0x7f405c006b40).stop 2023-11-15T06:45:50.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 msgr2=0x7f40681064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:50.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40681064e0 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4058013510 tx=0x7f405800b040).stop 2023-11-15T06:45:50.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 shutdown_connections 2023-11-15T06:45:50.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f405002ff60 0x7f4050032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:50.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 --2- 172.21.15.135:0/4129822727 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f40680f51a0 0x7f40681064e0 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:50.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 >> 172.21.15.135:0/4129822727 conn(0x7f40680f09a0 msgr2=0x7f406805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:50.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 shutdown_connections 2023-11-15T06:45:50.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:50.962+0000 7f4067fff700 1 -- 172.21.15.135:0/4129822727 wait complete. 2023-11-15T06:45:50.968 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:51.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:50.308918+0000 mgr.smithi123.razjtn (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:51.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:51 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:50.961652+0000 mon.smithi123 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.135:0/4129822727' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:52.339 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:52.340 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:53.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.487+0000 7fea8060c700 1 -- 172.21.15.135:0/2105170636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 msgr2=0x7fea780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:53.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.487+0000 7fea8060c700 1 --2- 172.21.15.135:0/2105170636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea780f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fea68009fb0 tx=0x7fea68009b30).stop 2023-11-15T06:45:53.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.487+0000 7fea8060c700 1 -- 172.21.15.135:0/2105170636 shutdown_connections 2023-11-15T06:45:53.492 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.487+0000 7fea8060c700 1 --2- 172.21.15.135:0/2105170636 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea780f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:53.493 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.487+0000 7fea8060c700 1 -- 172.21.15.135:0/2105170636 >> 172.21.15.135:0/2105170636 conn(0x7fea780f0b70 msgr2=0x7fea780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:53.493 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.487+0000 7fea8060c700 1 -- 172.21.15.135:0/2105170636 shutdown_connections 2023-11-15T06:45:53.493 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.487+0000 7fea8060c700 1 -- 172.21.15.135:0/2105170636 wait complete. 2023-11-15T06:45:53.493 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea8060c700 1 Processor -- start 2023-11-15T06:45:53.493 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea8060c700 1 -- start start 2023-11-15T06:45:53.494 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea8060c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea78105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:53.494 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea8060c700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fea78105b20 con 0x7fea780f5410 2023-11-15T06:45:53.494 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea7e3a8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea78105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:53.494 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea7e3a8700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea78105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:60038/0 (socket says 172.21.15.135:60038) 2023-11-15T06:45:53.494 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea7e3a8700 1 -- 172.21.15.135:0/2960616470 learned_addr learned my addr 172.21.15.135:0/2960616470 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:53.495 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea7e3a8700 1 -- 172.21.15.135:0/2960616470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fea68005040 con 0x7fea780f5410 2023-11-15T06:45:53.495 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea7e3a8700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea78105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fea68013510 tx=0x7fea6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:53.495 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.488+0000 7fea6effd700 1 -- 172.21.15.135:0/2960616470 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea68009320 con 0x7fea780f5410 2023-11-15T06:45:53.495 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.489+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fea78105d20 con 0x7fea780f5410 2023-11-15T06:45:53.495 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.489+0000 7fea6effd700 1 -- 172.21.15.135:0/2960616470 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fea68013f00 con 0x7fea780f5410 2023-11-15T06:45:53.496 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.489+0000 7fea6effd700 1 -- 172.21.15.135:0/2960616470 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fea68014d90 con 0x7fea780f5410 2023-11-15T06:45:53.496 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.489+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fea781061b0 con 0x7fea780f5410 2023-11-15T06:45:53.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.489+0000 7fea6effd700 1 -- 172.21.15.135:0/2960616470 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fea680095f0 con 0x7fea780f5410 2023-11-15T06:45:53.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.490+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fea78045ad0 con 0x7fea780f5410 2023-11-15T06:45:53.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.490+0000 7fea6effd700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fea6402ff60 0x7fea64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:53.497 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.490+0000 7fea6effd700 1 -- 172.21.15.135:0/2960616470 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fea68041000 con 0x7fea780f5410 2023-11-15T06:45:53.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.490+0000 7fea7dba7700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fea6402ff60 0x7fea64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:53.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.490+0000 7fea7dba7700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fea6402ff60 0x7fea64032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fea740096a0 tx=0x7fea74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:53.498 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.494+0000 7fea6effd700 1 -- 172.21.15.135:0/2960616470 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fea6801d7f0 con 0x7fea780f5410 2023-11-15T06:45:53.704 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.701+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fea780fae60 con 0x7fea780f5410 2023-11-15T06:45:53.704 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.701+0000 7fea6effd700 1 -- 172.21.15.135:0/2960616470 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fea68010230 con 0x7fea780f5410 2023-11-15T06:45:53.705 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:53.705 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:53.706 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fea6402ff60 msgr2=0x7fea64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:53.706 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fea6402ff60 0x7fea64032420 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fea740096a0 tx=0x7fea74006b40).stop 2023-11-15T06:45:53.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 msgr2=0x7fea78105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:53.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea78105630 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fea68013510 tx=0x7fea6800b040).stop 2023-11-15T06:45:53.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 shutdown_connections 2023-11-15T06:45:53.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fea6402ff60 0x7fea64032420 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:53.707 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 --2- 172.21.15.135:0/2960616470 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fea780f5410 0x7fea78105630 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:53.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 >> 172.21.15.135:0/2960616470 conn(0x7fea780f0b70 msgr2=0x7fea780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:53.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 shutdown_connections 2023-11-15T06:45:53.708 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:53.703+0000 7fea8060c700 1 -- 172.21.15.135:0/2960616470 wait complete. 2023-11-15T06:45:53.709 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:53.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:53 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:52.309244+0000 mgr.smithi123.razjtn (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:54.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:54 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:53.702689+0000 mon.smithi123 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.135:0/2960616470' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:55.039 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:55.040 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:55.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:55 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:54.309592+0000 mgr.smithi123.razjtn (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:56.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.158+0000 7fc8406dd700 1 -- 172.21.15.135:0/3589791925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 msgr2=0x7fc8380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:56.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.158+0000 7fc8406dd700 1 --2- 172.21.15.135:0/3589791925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8380f5830 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fc828009fb0 tx=0x7fc828009b30).stop 2023-11-15T06:45:56.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.158+0000 7fc8406dd700 1 -- 172.21.15.135:0/3589791925 shutdown_connections 2023-11-15T06:45:56.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.158+0000 7fc8406dd700 1 --2- 172.21.15.135:0/3589791925 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8380f5830 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:56.163 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.158+0000 7fc8406dd700 1 -- 172.21.15.135:0/3589791925 >> 172.21.15.135:0/3589791925 conn(0x7fc8380f0b70 msgr2=0x7fc8380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:56.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.158+0000 7fc8406dd700 1 -- 172.21.15.135:0/3589791925 shutdown_connections 2023-11-15T06:45:56.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.158+0000 7fc8406dd700 1 -- 172.21.15.135:0/3589791925 wait complete. 2023-11-15T06:45:56.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc8406dd700 1 Processor -- start 2023-11-15T06:45:56.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc8406dd700 1 -- start start 2023-11-15T06:45:56.164 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc8406dd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8381055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:56.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc8406dd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc838105ac0 con 0x7fc8380f5410 2023-11-15T06:45:56.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc83e479700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8381055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:56.165 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc83e479700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8381055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:60050/0 (socket says 172.21.15.135:60050) 2023-11-15T06:45:56.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc83e479700 1 -- 172.21.15.135:0/3417812383 learned_addr learned my addr 172.21.15.135:0/3417812383 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:56.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc83e479700 1 -- 172.21.15.135:0/3417812383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc828005040 con 0x7fc8380f5410 2023-11-15T06:45:56.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.159+0000 7fc83e479700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8381055d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc828013510 tx=0x7fc82800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:56.166 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.160+0000 7fc82effd700 1 -- 172.21.15.135:0/3417812383 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc828009320 con 0x7fc8380f5410 2023-11-15T06:45:56.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.160+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc838105cc0 con 0x7fc8380f5410 2023-11-15T06:45:56.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.160+0000 7fc82effd700 1 -- 172.21.15.135:0/3417812383 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc828013f00 con 0x7fc8380f5410 2023-11-15T06:45:56.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.160+0000 7fc82effd700 1 -- 172.21.15.135:0/3417812383 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc828014d90 con 0x7fc8380f5410 2023-11-15T06:45:56.167 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.160+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc838106150 con 0x7fc8380f5410 2023-11-15T06:45:56.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.160+0000 7fc82effd700 1 -- 172.21.15.135:0/3417812383 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc8280095f0 con 0x7fc8380f5410 2023-11-15T06:45:56.168 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.160+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc838045ad0 con 0x7fc8380f5410 2023-11-15T06:45:56.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.161+0000 7fc82effd700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc82402ffb0 0x7fc824032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:56.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.161+0000 7fc82effd700 1 -- 172.21.15.135:0/3417812383 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc828040fd0 con 0x7fc8380f5410 2023-11-15T06:45:56.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.161+0000 7fc83dc78700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc82402ffb0 0x7fc824032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:56.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.161+0000 7fc83dc78700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc82402ffb0 0x7fc824032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fc8340096a0 tx=0x7fc834006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:56.169 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.165+0000 7fc82effd700 1 -- 172.21.15.135:0/3417812383 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc828014420 con 0x7fc8380f5410 2023-11-15T06:45:56.375 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.372+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc838107000 con 0x7fc8380f5410 2023-11-15T06:45:56.376 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.373+0000 7fc82effd700 1 -- 172.21.15.135:0/3417812383 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc828010230 con 0x7fc8380f5410 2023-11-15T06:45:56.376 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:56.376 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:56.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc82402ffb0 msgr2=0x7fc824032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:56.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc82402ffb0 0x7fc824032470 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fc8340096a0 tx=0x7fc834006b40).stop 2023-11-15T06:45:56.379 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 msgr2=0x7fc8381055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:56.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8381055d0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fc828013510 tx=0x7fc82800b040).stop 2023-11-15T06:45:56.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 shutdown_connections 2023-11-15T06:45:56.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fc82402ffb0 0x7fc824032470 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:56.380 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 --2- 172.21.15.135:0/3417812383 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fc8380f5410 0x7fc8381055d0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:56.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.374+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 >> 172.21.15.135:0/3417812383 conn(0x7fc8380f0b70 msgr2=0x7fc8380f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:56.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.375+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 shutdown_connections 2023-11-15T06:45:56.381 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:56.375+0000 7fc8406dd700 1 -- 172.21.15.135:0/3417812383 wait complete. 2023-11-15T06:45:56.382 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:56.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:56 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:56.374058+0000 mon.smithi123 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.135:0/3417812383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:45:57.755 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:45:57.755 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:45:57.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:57 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:56.309962+0000 mgr.smithi123.razjtn (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:58.903 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.899+0000 7f4b73ebf700 1 -- 172.21.15.135:0/1813882128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 msgr2=0x7f4b6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:58.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.899+0000 7f4b73ebf700 1 --2- 172.21.15.135:0/1813882128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c0f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7f4b5c009fb0 tx=0x7f4b5c009b30).stop 2023-11-15T06:45:58.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.899+0000 7f4b73ebf700 1 -- 172.21.15.135:0/1813882128 shutdown_connections 2023-11-15T06:45:58.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.899+0000 7f4b73ebf700 1 --2- 172.21.15.135:0/1813882128 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c0f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:58.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.899+0000 7f4b73ebf700 1 -- 172.21.15.135:0/1813882128 >> 172.21.15.135:0/1813882128 conn(0x7f4b6c0f0b70 msgr2=0x7f4b6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:58.904 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.899+0000 7f4b73ebf700 1 -- 172.21.15.135:0/1813882128 shutdown_connections 2023-11-15T06:45:58.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.899+0000 7f4b73ebf700 1 -- 172.21.15.135:0/1813882128 wait complete. 2023-11-15T06:45:58.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b73ebf700 1 Processor -- start 2023-11-15T06:45:58.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b73ebf700 1 -- start start 2023-11-15T06:45:58.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b73ebf700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:58.905 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b73ebf700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b6c107240 con 0x7f4b6c0f5410 2023-11-15T06:45:58.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b71c5b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:58.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b71c5b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44562/0 (socket says 172.21.15.135:44562) 2023-11-15T06:45:58.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b71c5b700 1 -- 172.21.15.135:0/2143606043 learned_addr learned my addr 172.21.15.135:0/2143606043 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:45:58.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b71c5b700 1 -- 172.21.15.135:0/2143606043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b5c005040 con 0x7f4b6c0f5410 2023-11-15T06:45:58.906 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.900+0000 7f4b71c5b700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c10b850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4b5c013510 tx=0x7f4b5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:58.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.901+0000 7f4b627fc700 1 -- 172.21.15.135:0/2143606043 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b5c009320 con 0x7f4b6c0f5410 2023-11-15T06:45:58.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.901+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b6c060220 con 0x7f4b6c0f5410 2023-11-15T06:45:58.907 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.901+0000 7f4b627fc700 1 -- 172.21.15.135:0/2143606043 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b5c013f00 con 0x7f4b6c0f5410 2023-11-15T06:45:58.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.901+0000 7f4b627fc700 1 -- 172.21.15.135:0/2143606043 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b5c014d90 con 0x7f4b6c0f5410 2023-11-15T06:45:58.908 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.901+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b6c10bd40 con 0x7f4b6c0f5410 2023-11-15T06:45:58.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.901+0000 7f4b627fc700 1 -- 172.21.15.135:0/2143606043 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b5c0095f0 con 0x7f4b6c0f5410 2023-11-15T06:45:58.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.901+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b6c045ad0 con 0x7f4b6c0f5410 2023-11-15T06:45:58.909 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.902+0000 7f4b627fc700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4b5802ff60 0x7f4b58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:45:58.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.902+0000 7f4b627fc700 1 -- 172.21.15.135:0/2143606043 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b5c041090 con 0x7f4b6c0f5410 2023-11-15T06:45:58.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.902+0000 7f4b7145a700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4b5802ff60 0x7f4b58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:45:58.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.902+0000 7f4b7145a700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4b5802ff60 0x7f4b58032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4b680096a0 tx=0x7f4b68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:45:58.910 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:58.906+0000 7f4b627fc700 1 -- 172.21.15.135:0/2143606043 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b5c014420 con 0x7f4b6c0f5410 2023-11-15T06:45:59.114 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.111+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b6c0fae60 con 0x7f4b6c0f5410 2023-11-15T06:45:59.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.112+0000 7f4b627fc700 1 -- 172.21.15.135:0/2143606043 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4b5c01d7f0 con 0x7f4b6c0f5410 2023-11-15T06:45:59.115 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:45:59.116 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:45:59.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4b5802ff60 msgr2=0x7f4b58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:59.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4b5802ff60 0x7f4b58032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7f4b680096a0 tx=0x7f4b68006b40).stop 2023-11-15T06:45:59.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 msgr2=0x7f4b6c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:45:59.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c10b850 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7f4b5c013510 tx=0x7f4b5c00b040).stop 2023-11-15T06:45:59.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 shutdown_connections 2023-11-15T06:45:59.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f4b5802ff60 0x7f4b58032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:59.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 --2- 172.21.15.135:0/2143606043 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f4b6c0f5410 0x7f4b6c10b850 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:45:59.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 >> 172.21.15.135:0/2143606043 conn(0x7f4b6c0f0b70 msgr2=0x7f4b6c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:45:59.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 shutdown_connections 2023-11-15T06:45:59.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:45:59.114+0000 7f4b73ebf700 1 -- 172.21.15.135:0/2143606043 wait complete. 2023-11-15T06:45:59.120 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:45:59.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:45:58.310338+0000 mgr.smithi123.razjtn (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:45:59.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:45:59 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:45:59.113438+0000 mon.smithi123 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.135:0/2143606043' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:00.537 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:00.537 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:01.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 -- 172.21.15.135:0/2506071192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 msgr2=0x7f60480f8f20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:01.670 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 --2- 172.21.15.135:0/2506071192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f60480f8f20 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f6038009fb0 tx=0x7f6038009b30).stop 2023-11-15T06:46:01.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 -- 172.21.15.135:0/2506071192 shutdown_connections 2023-11-15T06:46:01.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 --2- 172.21.15.135:0/2506071192 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f60480f8f20 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:01.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 -- 172.21.15.135:0/2506071192 >> 172.21.15.135:0/2506071192 conn(0x7f60480f09a0 msgr2=0x7f60480f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:01.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 -- 172.21.15.135:0/2506071192 shutdown_connections 2023-11-15T06:46:01.671 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 -- 172.21.15.135:0/2506071192 wait complete. 2023-11-15T06:46:01.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 Processor -- start 2023-11-15T06:46:01.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 -- start start 2023-11-15T06:46:01.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f6048106500 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:01.672 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.667+0000 7f6047fff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60481069f0 con 0x7f60480601e0 2023-11-15T06:46:01.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.668+0000 7f6046ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f6048106500 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:01.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.668+0000 7f6046ffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f6048106500 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44578/0 (socket says 172.21.15.135:44578) 2023-11-15T06:46:01.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.668+0000 7f6046ffd700 1 -- 172.21.15.135:0/1405586978 learned_addr learned my addr 172.21.15.135:0/1405586978 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:01.674 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.668+0000 7f6046ffd700 1 -- 172.21.15.135:0/1405586978 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6038005040 con 0x7f60480601e0 2023-11-15T06:46:01.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.668+0000 7f6046ffd700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f6048106500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f6038013510 tx=0x7f603800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:01.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.668+0000 7f602f7fe700 1 -- 172.21.15.135:0/1405586978 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6038009320 con 0x7f60480601e0 2023-11-15T06:46:01.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.669+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6048106bf0 con 0x7f60480601e0 2023-11-15T06:46:01.675 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.669+0000 7f602f7fe700 1 -- 172.21.15.135:0/1405586978 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6038013f00 con 0x7f60480601e0 2023-11-15T06:46:01.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.669+0000 7f602f7fe700 1 -- 172.21.15.135:0/1405586978 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6038014d90 con 0x7f60480601e0 2023-11-15T06:46:01.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.669+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6048107080 con 0x7f60480601e0 2023-11-15T06:46:01.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.669+0000 7f602f7fe700 1 -- 172.21.15.135:0/1405586978 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60380095f0 con 0x7f60480601e0 2023-11-15T06:46:01.676 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.669+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6048045ad0 con 0x7f60480601e0 2023-11-15T06:46:01.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.669+0000 7f602f7fe700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f603002ff60 0x7f6030032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:01.677 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.670+0000 7f602f7fe700 1 -- 172.21.15.135:0/1405586978 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60380410c0 con 0x7f60480601e0 2023-11-15T06:46:01.679 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.670+0000 7f60467fc700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f603002ff60 0x7f6030032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:01.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.670+0000 7f60467fc700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f603002ff60 0x7f6030032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f603c0096a0 tx=0x7f603c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:01.680 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.674+0000 7f602f7fe700 1 -- 172.21.15.135:0/1405586978 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6038014420 con 0x7f60480601e0 2023-11-15T06:46:01.884 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.881+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6048107f30 con 0x7f60480601e0 2023-11-15T06:46:01.885 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.882+0000 7f602f7fe700 1 -- 172.21.15.135:0/1405586978 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6038010230 con 0x7f60480601e0 2023-11-15T06:46:01.885 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:01.886 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:01.886 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.883+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f603002ff60 msgr2=0x7f6030032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:01.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.883+0000 7f6047fff700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f603002ff60 0x7f6030032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f603c0096a0 tx=0x7f603c006b40).stop 2023-11-15T06:46:01.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.883+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 msgr2=0x7f6048106500 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:01.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.883+0000 7f6047fff700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f6048106500 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f6038013510 tx=0x7f603800b040).stop 2023-11-15T06:46:01.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.884+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 shutdown_connections 2023-11-15T06:46:01.887 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.884+0000 7f6047fff700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f603002ff60 0x7f6030032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:01.888 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.884+0000 7f6047fff700 1 --2- 172.21.15.135:0/1405586978 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f60480601e0 0x7f6048106500 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:01.888 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.884+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 >> 172.21.15.135:0/1405586978 conn(0x7f60480f09a0 msgr2=0x7f60480f1420 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:01.888 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.884+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 shutdown_connections 2023-11-15T06:46:01.888 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:01.884+0000 7f6047fff700 1 -- 172.21.15.135:0/1405586978 wait complete. 2023-11-15T06:46:01.889 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:01.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:01 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:00.310703+0000 mgr.smithi123.razjtn (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:02.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:02 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:01.882906+0000 mon.smithi123 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.135:0/1405586978' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:03.254 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:03.255 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:03.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:03 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:02.311005+0000 mgr.smithi123.razjtn (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:04.425 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2136307435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 msgr2=0x7fd4b00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:04.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 --2- 172.21.15.135:0/2136307435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b00f66c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7fd4ac009fb0 tx=0x7fd4ac009b30).stop 2023-11-15T06:46:04.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2136307435 shutdown_connections 2023-11-15T06:46:04.426 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 --2- 172.21.15.135:0/2136307435 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b00f66c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:04.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2136307435 >> 172.21.15.135:0/2136307435 conn(0x7fd4b00f1aa0 msgr2=0x7fd4b00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:04.427 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2136307435 shutdown_connections 2023-11-15T06:46:04.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2136307435 wait complete. 2023-11-15T06:46:04.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.419+0000 7fd4b73dd700 1 Processor -- start 2023-11-15T06:46:04.428 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b73dd700 1 -- start start 2023-11-15T06:46:04.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b73dd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:04.429 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b73dd700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4b01069a0 con 0x7fd4b00f62a0 2023-11-15T06:46:04.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b63db700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:04.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b63db700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44586/0 (socket says 172.21.15.135:44586) 2023-11-15T06:46:04.430 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b63db700 1 -- 172.21.15.135:0/2619091170 learned_addr learned my addr 172.21.15.135:0/2619091170 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:04.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b63db700 1 -- 172.21.15.135:0/2619091170 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd4ac005040 con 0x7fd4b00f62a0 2023-11-15T06:46:04.431 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4b63db700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b01064b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fd4ac013510 tx=0x7fd4ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:04.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.420+0000 7fd4a6ffd700 1 -- 172.21.15.135:0/2619091170 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4ac009320 con 0x7fd4b00f62a0 2023-11-15T06:46:04.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.421+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd4b0106ba0 con 0x7fd4b00f62a0 2023-11-15T06:46:04.432 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.421+0000 7fd4a6ffd700 1 -- 172.21.15.135:0/2619091170 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd4ac013f00 con 0x7fd4b00f62a0 2023-11-15T06:46:04.433 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.421+0000 7fd4a6ffd700 1 -- 172.21.15.135:0/2619091170 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd4ac014dc0 con 0x7fd4b00f62a0 2023-11-15T06:46:04.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.421+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd4b0107030 con 0x7fd4b00f62a0 2023-11-15T06:46:04.434 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.421+0000 7fd4a6ffd700 1 -- 172.21.15.135:0/2619091170 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4ac0095f0 con 0x7fd4b00f62a0 2023-11-15T06:46:04.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.422+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd4b0045ad0 con 0x7fd4b00f62a0 2023-11-15T06:46:04.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.422+0000 7fd4a6ffd700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd49c02ff60 0x7fd49c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:04.435 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.422+0000 7fd4a6ffd700 1 -- 172.21.15.135:0/2619091170 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd4ac041040 con 0x7fd4b00f62a0 2023-11-15T06:46:04.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.422+0000 7fd4b5bda700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd49c02ff60 0x7fd49c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:04.436 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.422+0000 7fd4b5bda700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd49c02ff60 0x7fd49c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fd4a00096a0 tx=0x7fd4a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:04.437 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.427+0000 7fd4a6ffd700 1 -- 172.21.15.135:0/2619091170 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd4ac014420 con 0x7fd4b00f62a0 2023-11-15T06:46:04.635 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.633+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd4b0107ee0 con 0x7fd4b00f62a0 2023-11-15T06:46:04.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.633+0000 7fd4a6ffd700 1 -- 172.21.15.135:0/2619091170 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd4ac010230 con 0x7fd4b00f62a0 2023-11-15T06:46:04.637 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:04.637 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:04.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.634+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd49c02ff60 msgr2=0x7fd49c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:04.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.634+0000 7fd4b73dd700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd49c02ff60 0x7fd49c032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fd4a00096a0 tx=0x7fd4a0006b40).stop 2023-11-15T06:46:04.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.634+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 msgr2=0x7fd4b01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:04.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.634+0000 7fd4b73dd700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b01064b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7fd4ac013510 tx=0x7fd4ac00b040).stop 2023-11-15T06:46:04.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.635+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 shutdown_connections 2023-11-15T06:46:04.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.635+0000 7fd4b73dd700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd49c02ff60 0x7fd49c032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:04.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.635+0000 7fd4b73dd700 1 --2- 172.21.15.135:0/2619091170 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd4b00f62a0 0x7fd4b01064b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:04.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.635+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 >> 172.21.15.135:0/2619091170 conn(0x7fd4b00f1aa0 msgr2=0x7fd4b00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:04.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.635+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 shutdown_connections 2023-11-15T06:46:04.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:04.635+0000 7fd4b73dd700 1 -- 172.21.15.135:0/2619091170 wait complete. 2023-11-15T06:46:04.640 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:05.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:04.311312+0000 mgr.smithi123.razjtn (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:05.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:05 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:04.634312+0000 mon.smithi123 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.135:0/2619091170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:06.095 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:06.096 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:07.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.269+0000 7efd1dd78700 1 -- 172.21.15.135:0/1741475296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 msgr2=0x7efd180f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:07.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.269+0000 7efd1dd78700 1 --2- 172.21.15.135:0/1741475296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd180f5830 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7efd00009fb0 tx=0x7efd00009b30).stop 2023-11-15T06:46:07.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.269+0000 7efd1dd78700 1 -- 172.21.15.135:0/1741475296 shutdown_connections 2023-11-15T06:46:07.273 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.269+0000 7efd1dd78700 1 --2- 172.21.15.135:0/1741475296 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd180f5830 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:07.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.269+0000 7efd1dd78700 1 -- 172.21.15.135:0/1741475296 >> 172.21.15.135:0/1741475296 conn(0x7efd180f0b70 msgr2=0x7efd180f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:07.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.269+0000 7efd1dd78700 1 -- 172.21.15.135:0/1741475296 shutdown_connections 2023-11-15T06:46:07.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.269+0000 7efd1dd78700 1 -- 172.21.15.135:0/1741475296 wait complete. 2023-11-15T06:46:07.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.270+0000 7efd1dd78700 1 Processor -- start 2023-11-15T06:46:07.274 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.270+0000 7efd1dd78700 1 -- start start 2023-11-15T06:46:07.275 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.270+0000 7efd1dd78700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd18105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:07.275 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.270+0000 7efd1dd78700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd18105af0 con 0x7efd180f5410 2023-11-15T06:46:07.277 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.270+0000 7efd177fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd18105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:07.277 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.270+0000 7efd177fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd18105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:44606/0 (socket says 172.21.15.135:44606) 2023-11-15T06:46:07.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.270+0000 7efd177fe700 1 -- 172.21.15.135:0/1593624015 learned_addr learned my addr 172.21.15.135:0/1593624015 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:07.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.271+0000 7efd177fe700 1 -- 172.21.15.135:0/1593624015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd00005040 con 0x7efd180f5410 2023-11-15T06:46:07.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.271+0000 7efd177fe700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd18105600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7efd00014000 tx=0x7efd0000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:07.278 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.271+0000 7efd0ffff700 1 -- 172.21.15.135:0/1593624015 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd000095f0 con 0x7efd180f5410 2023-11-15T06:46:07.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.271+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd18105cf0 con 0x7efd180f5410 2023-11-15T06:46:07.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.271+0000 7efd0ffff700 1 -- 172.21.15.135:0/1593624015 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd00007e70 con 0x7efd180f5410 2023-11-15T06:46:07.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.271+0000 7efd0ffff700 1 -- 172.21.15.135:0/1593624015 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd00014ea0 con 0x7efd180f5410 2023-11-15T06:46:07.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.271+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd18106180 con 0x7efd180f5410 2023-11-15T06:46:07.279 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.272+0000 7efd0ffff700 1 -- 172.21.15.135:0/1593624015 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd00014690 con 0x7efd180f5410 2023-11-15T06:46:07.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.272+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd18045ad0 con 0x7efd180f5410 2023-11-15T06:46:07.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.272+0000 7efd0ffff700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efd0402ff60 0x7efd04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:07.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.272+0000 7efd0ffff700 1 -- 172.21.15.135:0/1593624015 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd000408f0 con 0x7efd180f5410 2023-11-15T06:46:07.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.272+0000 7efd16ffd700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efd0402ff60 0x7efd04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:07.280 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.272+0000 7efd16ffd700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efd0402ff60 0x7efd04032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7efd080096a0 tx=0x7efd08006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:07.281 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.276+0000 7efd0ffff700 1 -- 172.21.15.135:0/1593624015 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd0000f300 con 0x7efd180f5410 2023-11-15T06:46:07.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.481+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd180fae60 con 0x7efd180f5410 2023-11-15T06:46:07.484 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.482+0000 7efd0ffff700 1 -- 172.21.15.135:0/1593624015 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efd0000f300 con 0x7efd180f5410 2023-11-15T06:46:07.485 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:07.485 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:07.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efd0402ff60 msgr2=0x7efd04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:07.486 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efd0402ff60 0x7efd04032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7efd080096a0 tx=0x7efd08006b40).stop 2023-11-15T06:46:07.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 msgr2=0x7efd18105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:07.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd18105600 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7efd00014000 tx=0x7efd0000b040).stop 2023-11-15T06:46:07.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 shutdown_connections 2023-11-15T06:46:07.487 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7efd0402ff60 0x7efd04032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:07.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 --2- 172.21.15.135:0/1593624015 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7efd180f5410 0x7efd18105600 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:07.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 >> 172.21.15.135:0/1593624015 conn(0x7efd180f0b70 msgr2=0x7efd180f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:07.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 shutdown_connections 2023-11-15T06:46:07.488 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:07.483+0000 7efd1dd78700 1 -- 172.21.15.135:0/1593624015 wait complete. 2023-11-15T06:46:07.489 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:07.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:06.311609+0000 mgr.smithi123.razjtn (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:07.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:07 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:07.482768+0000 mon.smithi123 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.135:0/1593624015' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:08.871 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:08.872 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:09.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:09 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:08.311862+0000 mgr.smithi123.razjtn (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:10.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 -- 172.21.15.135:0/1670530981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 msgr2=0x7fa5000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:10.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 --2- 172.21.15.135:0/1670530981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa5000f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7fa4f0009fb0 tx=0x7fa4f0009b30).stop 2023-11-15T06:46:10.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 -- 172.21.15.135:0/1670530981 shutdown_connections 2023-11-15T06:46:10.111 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 --2- 172.21.15.135:0/1670530981 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa5000f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:10.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 -- 172.21.15.135:0/1670530981 >> 172.21.15.135:0/1670530981 conn(0x7fa5000f0b70 msgr2=0x7fa5000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:10.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 -- 172.21.15.135:0/1670530981 shutdown_connections 2023-11-15T06:46:10.112 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 -- 172.21.15.135:0/1670530981 wait complete. 2023-11-15T06:46:10.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 Processor -- start 2023-11-15T06:46:10.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 -- start start 2023-11-15T06:46:10.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa500109400 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:10.115 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.108+0000 7fa505617700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa500060240 con 0x7fa5000f5410 2023-11-15T06:46:10.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4feffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa500109400 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:10.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4feffd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa500109400 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:40890/0 (socket says 172.21.15.135:40890) 2023-11-15T06:46:10.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4feffd700 1 -- 172.21.15.135:0/3487683715 learned_addr learned my addr 172.21.15.135:0/3487683715 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:10.116 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4feffd700 1 -- 172.21.15.135:0/3487683715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa4f0005040 con 0x7fa5000f5410 2023-11-15T06:46:10.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4feffd700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa500109400 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa4f0013510 tx=0x7fa4f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:10.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4f77fe700 1 -- 172.21.15.135:0/3487683715 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4f0009320 con 0x7fa5000f5410 2023-11-15T06:46:10.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa500060440 con 0x7fa5000f5410 2023-11-15T06:46:10.117 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4f77fe700 1 -- 172.21.15.135:0/3487683715 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa4f0013f00 con 0x7fa5000f5410 2023-11-15T06:46:10.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa4f77fe700 1 -- 172.21.15.135:0/3487683715 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa4f0014d90 con 0x7fa5000f5410 2023-11-15T06:46:10.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.109+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa500109ab0 con 0x7fa5000f5410 2023-11-15T06:46:10.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.110+0000 7fa4f77fe700 1 -- 172.21.15.135:0/3487683715 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa4f00095f0 con 0x7fa5000f5410 2023-11-15T06:46:10.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.110+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa500045ad0 con 0x7fa5000f5410 2023-11-15T06:46:10.118 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.110+0000 7fa4f77fe700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa4ec02ff60 0x7fa4ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:10.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.110+0000 7fa4f77fe700 1 -- 172.21.15.135:0/3487683715 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa4f0041080 con 0x7fa5000f5410 2023-11-15T06:46:10.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.110+0000 7fa4fe7fc700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa4ec02ff60 0x7fa4ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:10.119 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.111+0000 7fa4fe7fc700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa4ec02ff60 0x7fa4ec032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa4e80096a0 tx=0x7fa4e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:10.120 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.115+0000 7fa4f77fe700 1 -- 172.21.15.135:0/3487683715 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa4f0014420 con 0x7fa5000f5410 2023-11-15T06:46:10.323 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.320+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5000fae60 con 0x7fa5000f5410 2023-11-15T06:46:10.323 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.320+0000 7fa4f77fe700 1 -- 172.21.15.135:0/3487683715 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa4f0010230 con 0x7fa5000f5410 2023-11-15T06:46:10.324 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:10.324 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:10.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa4ec02ff60 msgr2=0x7fa4ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:10.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa4ec02ff60 0x7fa4ec032420 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7fa4e80096a0 tx=0x7fa4e8006b40).stop 2023-11-15T06:46:10.325 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 msgr2=0x7fa500109400 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:10.326 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa500109400 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7fa4f0013510 tx=0x7fa4f000b040).stop 2023-11-15T06:46:10.326 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 shutdown_connections 2023-11-15T06:46:10.326 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fa4ec02ff60 0x7fa4ec032420 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:10.326 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 --2- 172.21.15.135:0/3487683715 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fa5000f5410 0x7fa500109400 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:10.327 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 >> 172.21.15.135:0/3487683715 conn(0x7fa5000f0b70 msgr2=0x7fa5000f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:10.327 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 shutdown_connections 2023-11-15T06:46:10.327 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:10.322+0000 7fa505617700 1 -- 172.21.15.135:0/3487683715 wait complete. 2023-11-15T06:46:10.327 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:10.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:10 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:10.321501+0000 mon.smithi123 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.135:0/3487683715' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:11.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:11.691 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:11.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:11 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:10.312172+0000 mgr.smithi123.razjtn (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:12.856 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.852+0000 7fd0a1e77700 1 -- 172.21.15.135:0/3234315531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 msgr2=0x7fd09c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:12.856 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.852+0000 7fd0a1e77700 1 --2- 172.21.15.135:0/3234315531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c0f6930 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7fd094009960 tx=0x7fd094004d10).stop 2023-11-15T06:46:12.856 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.852+0000 7fd0a1e77700 1 -- 172.21.15.135:0/3234315531 shutdown_connections 2023-11-15T06:46:12.856 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.852+0000 7fd0a1e77700 1 --2- 172.21.15.135:0/3234315531 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c0f6930 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:12.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.852+0000 7fd0a1e77700 1 -- 172.21.15.135:0/3234315531 >> 172.21.15.135:0/3234315531 conn(0x7fd09c0f1c70 msgr2=0x7fd09c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:12.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.852+0000 7fd0a1e77700 1 -- 172.21.15.135:0/3234315531 shutdown_connections 2023-11-15T06:46:12.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.852+0000 7fd0a1e77700 1 -- 172.21.15.135:0/3234315531 wait complete. 2023-11-15T06:46:12.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.853+0000 7fd0a1e77700 1 Processor -- start 2023-11-15T06:46:12.857 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.853+0000 7fd0a1e77700 1 -- start start 2023-11-15T06:46:12.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.853+0000 7fd0a1e77700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c108910 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:12.858 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.853+0000 7fd0a1e77700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd09c108e00 con 0x7fd09c0f6510 2023-11-15T06:46:12.860 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.853+0000 7fd09b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c108910 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:12.860 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.853+0000 7fd09b7fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c108910 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:40910/0 (socket says 172.21.15.135:40910) 2023-11-15T06:46:12.860 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.853+0000 7fd09b7fe700 1 -- 172.21.15.135:0/1175908670 learned_addr learned my addr 172.21.15.135:0/1175908670 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:12.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.854+0000 7fd09b7fe700 1 -- 172.21.15.135:0/1175908670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd094009640 con 0x7fd09c0f6510 2023-11-15T06:46:12.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.854+0000 7fd09b7fe700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c108910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fd094004b70 tx=0x7fd094004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:12.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.854+0000 7fd083fff700 1 -- 172.21.15.135:0/1175908670 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd094018070 con 0x7fd09c0f6510 2023-11-15T06:46:12.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.854+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd09c109000 con 0x7fd09c0f6510 2023-11-15T06:46:12.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.854+0000 7fd083fff700 1 -- 172.21.15.135:0/1175908670 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd09400b870 con 0x7fd09c0f6510 2023-11-15T06:46:12.861 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.854+0000 7fd083fff700 1 -- 172.21.15.135:0/1175908670 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd09401c560 con 0x7fd09c0f6510 2023-11-15T06:46:12.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.854+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd09c109490 con 0x7fd09c0f6510 2023-11-15T06:46:12.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.855+0000 7fd083fff700 1 -- 172.21.15.135:0/1175908670 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd094013630 con 0x7fd09c0f6510 2023-11-15T06:46:12.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.855+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd09c045ad0 con 0x7fd09c0f6510 2023-11-15T06:46:12.862 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.855+0000 7fd083fff700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd08402ff60 0x7fd084032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:12.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.855+0000 7fd083fff700 1 -- 172.21.15.135:0/1175908670 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd094041280 con 0x7fd09c0f6510 2023-11-15T06:46:12.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.855+0000 7fd09affd700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd08402ff60 0x7fd084032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:12.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.856+0000 7fd09affd700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd08402ff60 0x7fd084032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd0900096a0 tx=0x7fd090006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:12.863 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:12.860+0000 7fd083fff700 1 -- 172.21.15.135:0/1175908670 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd094004810 con 0x7fd09c0f6510 2023-11-15T06:46:13.066 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.063+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd09c10a2e0 con 0x7fd09c0f6510 2023-11-15T06:46:13.067 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.064+0000 7fd083fff700 1 -- 172.21.15.135:0/1175908670 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd094011320 con 0x7fd09c0f6510 2023-11-15T06:46:13.067 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:13.067 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:13.068 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.065+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd08402ff60 msgr2=0x7fd084032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:13.069 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.065+0000 7fd0a1e77700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd08402ff60 0x7fd084032420 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7fd0900096a0 tx=0x7fd090006b40).stop 2023-11-15T06:46:13.069 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.065+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 msgr2=0x7fd09c108910 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:13.069 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.065+0000 7fd0a1e77700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c108910 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7fd094004b70 tx=0x7fd094004d10).stop 2023-11-15T06:46:13.069 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.065+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 shutdown_connections 2023-11-15T06:46:13.069 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.065+0000 7fd0a1e77700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fd08402ff60 0x7fd084032420 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:13.070 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.065+0000 7fd0a1e77700 1 --2- 172.21.15.135:0/1175908670 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fd09c0f6510 0x7fd09c108910 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:13.070 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.066+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 >> 172.21.15.135:0/1175908670 conn(0x7fd09c0f1c70 msgr2=0x7fd09c0f2720 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:13.070 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.066+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 shutdown_connections 2023-11-15T06:46:13.070 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:13.066+0000 7fd0a1e77700 1 -- 172.21.15.135:0/1175908670 wait complete. 2023-11-15T06:46:13.071 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:13.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:12.312509+0000 mgr.smithi123.razjtn (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:13.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:13 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:13.065109+0000 mon.smithi123 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.135:0/1175908670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:14.454 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:14.454 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:15.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 -- 172.21.15.135:0/3027787493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 msgr2=0x7ffb680f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:15.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 --2- 172.21.15.135:0/3027787493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb680f4b60 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7ffb5c009fb0 tx=0x7ffb5c009b30).stop 2023-11-15T06:46:15.634 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 -- 172.21.15.135:0/3027787493 shutdown_connections 2023-11-15T06:46:15.635 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 --2- 172.21.15.135:0/3027787493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb680f4b60 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:15.635 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 -- 172.21.15.135:0/3027787493 >> 172.21.15.135:0/3027787493 conn(0x7ffb680f0260 msgr2=0x7ffb680f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:15.635 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 -- 172.21.15.135:0/3027787493 shutdown_connections 2023-11-15T06:46:15.636 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 -- 172.21.15.135:0/3027787493 wait complete. 2023-11-15T06:46:15.638 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 Processor -- start 2023-11-15T06:46:15.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.631+0000 7ffb6ca43700 1 -- start start 2023-11-15T06:46:15.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.632+0000 7ffb6ca43700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb681066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:15.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.632+0000 7ffb6ca43700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb68106bc0 con 0x7ffb680f4740 2023-11-15T06:46:15.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.632+0000 7ffb6659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb681066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:15.639 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.632+0000 7ffb6659c700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb681066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:40914/0 (socket says 172.21.15.135:40914) 2023-11-15T06:46:15.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.632+0000 7ffb6659c700 1 -- 172.21.15.135:0/1353566877 learned_addr learned my addr 172.21.15.135:0/1353566877 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:15.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.632+0000 7ffb6659c700 1 -- 172.21.15.135:0/1353566877 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb5c005040 con 0x7ffb680f4740 2023-11-15T06:46:15.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.632+0000 7ffb6659c700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb681066d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ffb5c013510 tx=0x7ffb5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:15.640 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.633+0000 7ffb56ffd700 1 -- 172.21.15.135:0/1353566877 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c009320 con 0x7ffb680f4740 2023-11-15T06:46:15.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.633+0000 7ffb56ffd700 1 -- 172.21.15.135:0/1353566877 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb5c013f00 con 0x7ffb680f4740 2023-11-15T06:46:15.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.633+0000 7ffb56ffd700 1 -- 172.21.15.135:0/1353566877 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb5c014d90 con 0x7ffb680f4740 2023-11-15T06:46:15.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.633+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb68106dc0 con 0x7ffb680f4740 2023-11-15T06:46:15.641 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.633+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb68107250 con 0x7ffb680f4740 2023-11-15T06:46:15.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.633+0000 7ffb56ffd700 1 -- 172.21.15.135:0/1353566877 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb5c0095f0 con 0x7ffb680f4740 2023-11-15T06:46:15.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.633+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb68045ad0 con 0x7ffb680f4740 2023-11-15T06:46:15.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.634+0000 7ffb56ffd700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5002ff60 0x7ffb50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:15.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.634+0000 7ffb56ffd700 1 -- 172.21.15.135:0/1353566877 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb5c041000 con 0x7ffb680f4740 2023-11-15T06:46:15.642 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.634+0000 7ffb65d9b700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5002ff60 0x7ffb50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:15.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.634+0000 7ffb65d9b700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5002ff60 0x7ffb50032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ffb580096a0 tx=0x7ffb58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:15.643 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.638+0000 7ffb56ffd700 1 -- 172.21.15.135:0/1353566877 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb5c01d7f0 con 0x7ffb680f4740 2023-11-15T06:46:15.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.845+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb68108100 con 0x7ffb680f4740 2023-11-15T06:46:15.848 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.846+0000 7ffb56ffd700 1 -- 172.21.15.135:0/1353566877 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb5c010230 con 0x7ffb680f4740 2023-11-15T06:46:15.849 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:15.849 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:15.850 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5002ff60 msgr2=0x7ffb50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:15.851 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5002ff60 0x7ffb50032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7ffb580096a0 tx=0x7ffb58006b40).stop 2023-11-15T06:46:15.851 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 msgr2=0x7ffb681066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:15.851 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb681066d0 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7ffb5c013510 tx=0x7ffb5c00b040).stop 2023-11-15T06:46:15.851 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 shutdown_connections 2023-11-15T06:46:15.852 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7ffb5002ff60 0x7ffb50032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:15.852 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 --2- 172.21.15.135:0/1353566877 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7ffb680f4740 0x7ffb681066d0 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:15.852 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 >> 172.21.15.135:0/1353566877 conn(0x7ffb680f0260 msgr2=0x7ffb680f0d80 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:15.852 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 shutdown_connections 2023-11-15T06:46:15.853 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:15.847+0000 7ffb6ca43700 1 -- 172.21.15.135:0/1353566877 wait complete. 2023-11-15T06:46:15.853 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:15.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:15 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:14.312909+0000 mgr.smithi123.razjtn (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:16.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:16 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:15.846646+0000 mon.smithi123 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.135:0/1353566877' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:17.237 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:17.237 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:17.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:17 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:16.313306+0000 mgr.smithi123.razjtn (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:18.382 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 -- 172.21.15.135:0/925945410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 msgr2=0x7f84ac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:18.382 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 --2- 172.21.15.135:0/925945410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac0f55c0 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f849c009fb0 tx=0x7f849c009b30).stop 2023-11-15T06:46:18.382 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 -- 172.21.15.135:0/925945410 shutdown_connections 2023-11-15T06:46:18.382 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 --2- 172.21.15.135:0/925945410 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac0f55c0 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:18.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 -- 172.21.15.135:0/925945410 >> 172.21.15.135:0/925945410 conn(0x7f84ac0f09a0 msgr2=0x7f84ac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:18.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 -- 172.21.15.135:0/925945410 shutdown_connections 2023-11-15T06:46:18.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 -- 172.21.15.135:0/925945410 wait complete. 2023-11-15T06:46:18.383 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 Processor -- start 2023-11-15T06:46:18.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.377+0000 7f84b2507700 1 -- start start 2023-11-15T06:46:18.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.378+0000 7f84b2507700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac1053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:18.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.378+0000 7f84b2507700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f84ac1058a0 con 0x7f84ac0f51a0 2023-11-15T06:46:18.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.378+0000 7f84b1505700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac1053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:18.384 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.378+0000 7f84b1505700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac1053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53200/0 (socket says 172.21.15.135:53200) 2023-11-15T06:46:18.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.378+0000 7f84b1505700 1 -- 172.21.15.135:0/2844192229 learned_addr learned my addr 172.21.15.135:0/2844192229 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:18.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.378+0000 7f84b1505700 1 -- 172.21.15.135:0/2844192229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f849c005040 con 0x7f84ac0f51a0 2023-11-15T06:46:18.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.378+0000 7f84b1505700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac1053b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f849c013510 tx=0x7f849c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:18.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.379+0000 7f84a1ffb700 1 -- 172.21.15.135:0/2844192229 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849c009320 con 0x7f84ac0f51a0 2023-11-15T06:46:18.385 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.379+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f84ac105aa0 con 0x7f84ac0f51a0 2023-11-15T06:46:18.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.379+0000 7f84a1ffb700 1 -- 172.21.15.135:0/2844192229 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f849c013f00 con 0x7f84ac0f51a0 2023-11-15T06:46:18.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.379+0000 7f84a1ffb700 1 -- 172.21.15.135:0/2844192229 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f849c014dc0 con 0x7f84ac0f51a0 2023-11-15T06:46:18.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.379+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f84ac105f30 con 0x7f84ac0f51a0 2023-11-15T06:46:18.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.380+0000 7f84a1ffb700 1 -- 172.21.15.135:0/2844192229 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f849c0095f0 con 0x7f84ac0f51a0 2023-11-15T06:46:18.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.380+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f84ac045ad0 con 0x7f84ac0f51a0 2023-11-15T06:46:18.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.380+0000 7f84a1ffb700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f849802ffb0 0x7f8498032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:18.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.380+0000 7f84a1ffb700 1 -- 172.21.15.135:0/2844192229 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f849c0410e0 con 0x7f84ac0f51a0 2023-11-15T06:46:18.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.380+0000 7f84b0d04700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f849802ffb0 0x7f8498032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:18.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.381+0000 7f84b0d04700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f849802ffb0 0x7f8498032470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f84a80096a0 tx=0x7f84a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:18.388 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.385+0000 7f84a1ffb700 1 -- 172.21.15.135:0/2844192229 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f849c014420 con 0x7f84ac0f51a0 2023-11-15T06:46:18.594 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.590+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f84ac106d80 con 0x7f84ac0f51a0 2023-11-15T06:46:18.594 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.591+0000 7f84a1ffb700 1 -- 172.21.15.135:0/2844192229 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f849c01c920 con 0x7f84ac0f51a0 2023-11-15T06:46:18.595 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:18.595 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:18.596 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.592+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f849802ffb0 msgr2=0x7f8498032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:18.596 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.592+0000 7f84b2507700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f849802ffb0 0x7f8498032470 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f84a80096a0 tx=0x7f84a8006b40).stop 2023-11-15T06:46:18.596 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.592+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 msgr2=0x7f84ac1053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:18.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.592+0000 7f84b2507700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac1053b0 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f849c013510 tx=0x7f849c00b040).stop 2023-11-15T06:46:18.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.592+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 shutdown_connections 2023-11-15T06:46:18.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.592+0000 7f84b2507700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f849802ffb0 0x7f8498032470 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:18.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.592+0000 7f84b2507700 1 --2- 172.21.15.135:0/2844192229 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f84ac0f51a0 0x7f84ac1053b0 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:18.597 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.593+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 >> 172.21.15.135:0/2844192229 conn(0x7f84ac0f09a0 msgr2=0x7f84ac0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:18.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.593+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 shutdown_connections 2023-11-15T06:46:18.598 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:18.593+0000 7f84b2507700 1 -- 172.21.15.135:0/2844192229 wait complete. 2023-11-15T06:46:18.598 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:19.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:18.313591+0000 mgr.smithi123.razjtn (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:19.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:19 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:18.592190+0000 mon.smithi123 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.135:0/2844192229' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:19.992 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:19.993 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:21.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.171+0000 7f613ffff700 1 -- 172.21.15.135:0/1104973437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 msgr2=0x7f61400f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:21.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.171+0000 7f613ffff700 1 --2- 172.21.15.135:0/1104973437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61400f55c0 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f6134009fb0 tx=0x7f6134009b30).stop 2023-11-15T06:46:21.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.171+0000 7f613ffff700 1 -- 172.21.15.135:0/1104973437 shutdown_connections 2023-11-15T06:46:21.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.171+0000 7f613ffff700 1 --2- 172.21.15.135:0/1104973437 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61400f55c0 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:21.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.171+0000 7f613ffff700 1 -- 172.21.15.135:0/1104973437 >> 172.21.15.135:0/1104973437 conn(0x7f61400f09a0 msgr2=0x7f61400f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:21.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.171+0000 7f613ffff700 1 -- 172.21.15.135:0/1104973437 shutdown_connections 2023-11-15T06:46:21.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.171+0000 7f613ffff700 1 -- 172.21.15.135:0/1104973437 wait complete. 2023-11-15T06:46:21.177 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613ffff700 1 Processor -- start 2023-11-15T06:46:21.177 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613ffff700 1 -- start start 2023-11-15T06:46:21.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613ffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61401053e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:21.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613ffff700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f61401058d0 con 0x7f61400f51a0 2023-11-15T06:46:21.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61401053e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:21.178 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613effd700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61401053e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53218/0 (socket says 172.21.15.135:53218) 2023-11-15T06:46:21.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613effd700 1 -- 172.21.15.135:0/1271527213 learned_addr learned my addr 172.21.15.135:0/1271527213 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:21.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.172+0000 7f613effd700 1 -- 172.21.15.135:0/1271527213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6134005040 con 0x7f61400f51a0 2023-11-15T06:46:21.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.173+0000 7f613effd700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61401053e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f6134009830 tx=0x7f613400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:21.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.173+0000 7f61277fe700 1 -- 172.21.15.135:0/1271527213 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6134009320 con 0x7f61400f51a0 2023-11-15T06:46:21.179 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.173+0000 7f61277fe700 1 -- 172.21.15.135:0/1271527213 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f61340136c0 con 0x7f61400f51a0 2023-11-15T06:46:21.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.173+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6140105ad0 con 0x7f61400f51a0 2023-11-15T06:46:21.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.173+0000 7f61277fe700 1 -- 172.21.15.135:0/1271527213 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f613401ee90 con 0x7f61400f51a0 2023-11-15T06:46:21.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.173+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6140105f60 con 0x7f61400f51a0 2023-11-15T06:46:21.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.174+0000 7f61277fe700 1 -- 172.21.15.135:0/1271527213 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f613401e420 con 0x7f61400f51a0 2023-11-15T06:46:21.180 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.174+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6140045ad0 con 0x7f61400f51a0 2023-11-15T06:46:21.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.174+0000 7f61277fe700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f612802ff10 0x7f61280323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:21.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.174+0000 7f61277fe700 1 -- 172.21.15.135:0/1271527213 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6134040f40 con 0x7f61400f51a0 2023-11-15T06:46:21.181 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.174+0000 7f613e7fc700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f612802ff10 0x7f61280323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:21.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.174+0000 7f613e7fc700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f612802ff10 0x7f61280323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f61300096a0 tx=0x7f6130006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:21.182 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.179+0000 7f61277fe700 1 -- 172.21.15.135:0/1271527213 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f613406d2b0 con 0x7f61400f51a0 2023-11-15T06:46:21.386 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.383+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6140106e10 con 0x7f61400f51a0 2023-11-15T06:46:21.387 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.383+0000 7f61277fe700 1 -- 172.21.15.135:0/1271527213 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f61340095f0 con 0x7f61400f51a0 2023-11-15T06:46:21.387 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:21.387 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:21.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f612802ff10 msgr2=0x7f61280323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:21.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f612802ff10 0x7f61280323d0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f61300096a0 tx=0x7f6130006b40).stop 2023-11-15T06:46:21.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 msgr2=0x7f61401053e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:21.389 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61401053e0 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f6134009830 tx=0x7f613400b040).stop 2023-11-15T06:46:21.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 shutdown_connections 2023-11-15T06:46:21.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f612802ff10 0x7f61280323d0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:21.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 --2- 172.21.15.135:0/1271527213 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f61400f51a0 0x7f61401053e0 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:21.390 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 >> 172.21.15.135:0/1271527213 conn(0x7f61400f09a0 msgr2=0x7f61400f13e0 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:21.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 shutdown_connections 2023-11-15T06:46:21.391 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:21.385+0000 7f613ffff700 1 -- 172.21.15.135:0/1271527213 wait complete. 2023-11-15T06:46:21.391 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:21.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:20.313929+0000 mgr.smithi123.razjtn (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:21.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:21 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:21.384598+0000 mon.smithi123 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.135:0/1271527213' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:22.772 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:22.773 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:23.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:23 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:22.314349+0000 mgr.smithi123.razjtn (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:23.946 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.941+0000 7f3c0debb700 1 -- 172.21.15.135:0/3480767478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 msgr2=0x7f3c080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:23.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.941+0000 7f3c0debb700 1 --2- 172.21.15.135:0/3480767478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c080f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f3bf8009fb0 tx=0x7f3bf8009b30).stop 2023-11-15T06:46:23.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.941+0000 7f3c0debb700 1 -- 172.21.15.135:0/3480767478 shutdown_connections 2023-11-15T06:46:23.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.941+0000 7f3c0debb700 1 --2- 172.21.15.135:0/3480767478 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c080f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:23.947 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.941+0000 7f3c0debb700 1 -- 172.21.15.135:0/3480767478 >> 172.21.15.135:0/3480767478 conn(0x7f3c080f0b70 msgr2=0x7f3c080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:23.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.941+0000 7f3c0debb700 1 -- 172.21.15.135:0/3480767478 shutdown_connections 2023-11-15T06:46:23.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.942+0000 7f3c0debb700 1 -- 172.21.15.135:0/3480767478 wait complete. 2023-11-15T06:46:23.949 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.942+0000 7f3c0debb700 1 Processor -- start 2023-11-15T06:46:23.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.942+0000 7f3c0debb700 1 -- start start 2023-11-15T06:46:23.950 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.942+0000 7f3c0debb700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c08108920 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:23.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.942+0000 7f3c0debb700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c08108e10 con 0x7f3c080f5410 2023-11-15T06:46:23.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3c077fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c08108920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:23.951 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3c077fe700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c08108920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53232/0 (socket says 172.21.15.135:53232) 2023-11-15T06:46:23.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3c077fe700 1 -- 172.21.15.135:0/1004472363 learned_addr learned my addr 172.21.15.135:0/1004472363 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:23.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3c077fe700 1 -- 172.21.15.135:0/1004472363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3bf8005040 con 0x7f3c080f5410 2023-11-15T06:46:23.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3c077fe700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c08108920 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f3bf8013510 tx=0x7f3bf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:23.952 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3bfffff700 1 -- 172.21.15.135:0/1004472363 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bf8009320 con 0x7f3c080f5410 2023-11-15T06:46:23.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3bfffff700 1 -- 172.21.15.135:0/1004472363 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3bf8013f00 con 0x7f3c080f5410 2023-11-15T06:46:23.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c08109010 con 0x7f3c080f5410 2023-11-15T06:46:23.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3bfffff700 1 -- 172.21.15.135:0/1004472363 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3bf8014d90 con 0x7f3c080f5410 2023-11-15T06:46:23.953 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.943+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c081094a0 con 0x7f3c080f5410 2023-11-15T06:46:23.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.944+0000 7f3bfffff700 1 -- 172.21.15.135:0/1004472363 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3bf8014420 con 0x7f3c080f5410 2023-11-15T06:46:23.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.944+0000 7f3bfffff700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bf402ff60 0x7f3bf4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:23.955 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.944+0000 7f3bfffff700 1 -- 172.21.15.135:0/1004472363 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3bf8041000 con 0x7f3c080f5410 2023-11-15T06:46:23.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.944+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c08045ad0 con 0x7f3c080f5410 2023-11-15T06:46:23.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.944+0000 7f3c06ffd700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bf402ff60 0x7f3bf4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:23.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.945+0000 7f3c06ffd700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bf402ff60 0x7f3bf4032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3bf00096a0 tx=0x7f3bf0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:23.956 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:23.949+0000 7f3bfffff700 1 -- 172.21.15.135:0/1004472363 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3bf80095f0 con 0x7f3c080f5410 2023-11-15T06:46:24.158 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.155+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c08102550 con 0x7f3c080f5410 2023-11-15T06:46:24.158 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.156+0000 7f3bfffff700 1 -- 172.21.15.135:0/1004472363 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3bf801db70 con 0x7f3c080f5410 2023-11-15T06:46:24.159 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:24.159 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:24.160 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bf402ff60 msgr2=0x7f3bf4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:24.160 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bf402ff60 0x7f3bf4032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f3bf00096a0 tx=0x7f3bf0006b40).stop 2023-11-15T06:46:24.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 msgr2=0x7f3c08108920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:24.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c08108920 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f3bf8013510 tx=0x7f3bf800b040).stop 2023-11-15T06:46:24.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 shutdown_connections 2023-11-15T06:46:24.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3bf402ff60 0x7f3bf4032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:24.161 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 --2- 172.21.15.135:0/1004472363 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3c080f5410 0x7f3c08108920 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:24.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 >> 172.21.15.135:0/1004472363 conn(0x7f3c080f0b70 msgr2=0x7f3c080f2f60 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:24.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 shutdown_connections 2023-11-15T06:46:24.162 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:24.157+0000 7f3c0debb700 1 -- 172.21.15.135:0/1004472363 wait complete. 2023-11-15T06:46:24.163 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:24.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:24 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:24.156897+0000 mon.smithi123 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.135:0/1004472363' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:25.554 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:25.554 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:25.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:25 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:24.314720+0000 mgr.smithi123.razjtn (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:26.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.711+0000 7f3edf739700 1 -- 172.21.15.135:0/2251810217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 msgr2=0x7f3ed80f5d70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:26.715 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.711+0000 7f3edf739700 1 --2- 172.21.15.135:0/2251810217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed80f5d70 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f3ec8009fb0 tx=0x7f3ec8009b30).stop 2023-11-15T06:46:26.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.711+0000 7f3edf739700 1 -- 172.21.15.135:0/2251810217 shutdown_connections 2023-11-15T06:46:26.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.711+0000 7f3edf739700 1 --2- 172.21.15.135:0/2251810217 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed80f5d70 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:26.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.711+0000 7f3edf739700 1 -- 172.21.15.135:0/2251810217 >> 172.21.15.135:0/2251810217 conn(0x7f3ed80f0260 msgr2=0x7f3ed80f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:26.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.711+0000 7f3edf739700 1 -- 172.21.15.135:0/2251810217 shutdown_connections 2023-11-15T06:46:26.716 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.711+0000 7f3edf739700 1 -- 172.21.15.135:0/2251810217 wait complete. 2023-11-15T06:46:26.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.712+0000 7f3edf739700 1 Processor -- start 2023-11-15T06:46:26.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.712+0000 7f3edf739700 1 -- start start 2023-11-15T06:46:26.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.712+0000 7f3edf739700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed8109410 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:26.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.712+0000 7f3edf739700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3ed80f5330 con 0x7f3ed80f5950 2023-11-15T06:46:26.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.712+0000 7f3edd4d5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed8109410 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:26.717 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.712+0000 7f3edd4d5700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed8109410 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:53244/0 (socket says 172.21.15.135:53244) 2023-11-15T06:46:26.718 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.712+0000 7f3edd4d5700 1 -- 172.21.15.135:0/534971326 learned_addr learned my addr 172.21.15.135:0/534971326 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:26.719 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.713+0000 7f3edd4d5700 1 -- 172.21.15.135:0/534971326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ec8005040 con 0x7f3ed80f5950 2023-11-15T06:46:26.719 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.713+0000 7f3edd4d5700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed8109410 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f3ec8013510 tx=0x7f3ec800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:26.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.713+0000 7f3ecdffb700 1 -- 172.21.15.135:0/534971326 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec8009320 con 0x7f3ed80f5950 2023-11-15T06:46:26.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.713+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3ed80f5530 con 0x7f3ed80f5950 2023-11-15T06:46:26.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.713+0000 7f3ecdffb700 1 -- 172.21.15.135:0/534971326 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ec8013f00 con 0x7f3ed80f5950 2023-11-15T06:46:26.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.713+0000 7f3ecdffb700 1 -- 172.21.15.135:0/534971326 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ec8014d90 con 0x7f3ed80f5950 2023-11-15T06:46:26.720 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.713+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3ed8109ac0 con 0x7f3ed80f5950 2023-11-15T06:46:26.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.714+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3ed8045ad0 con 0x7f3ed80f5950 2023-11-15T06:46:26.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.714+0000 7f3ecdffb700 1 -- 172.21.15.135:0/534971326 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ec80095f0 con 0x7f3ed80f5950 2023-11-15T06:46:26.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.714+0000 7f3ecdffb700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3ec402ff60 0x7f3ec4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:26.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.714+0000 7f3ecdffb700 1 -- 172.21.15.135:0/534971326 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ec8041080 con 0x7f3ed80f5950 2023-11-15T06:46:26.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.714+0000 7f3edccd4700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3ec402ff60 0x7f3ec4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:26.721 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.715+0000 7f3edccd4700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3ec402ff60 0x7f3ec4032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3ed40096a0 tx=0x7f3ed4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:26.722 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.719+0000 7f3ecdffb700 1 -- 172.21.15.135:0/534971326 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ec8014420 con 0x7f3ed80f5950 2023-11-15T06:46:26.928 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.926+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3ed810a2e0 con 0x7f3ed80f5950 2023-11-15T06:46:26.929 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.926+0000 7f3ecdffb700 1 -- 172.21.15.135:0/534971326 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ec801d7f0 con 0x7f3ed80f5950 2023-11-15T06:46:26.930 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:26.930 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:26.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3ec402ff60 msgr2=0x7f3ec4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:26.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3ec402ff60 0x7f3ec4032420 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f3ed40096a0 tx=0x7f3ed4006b40).stop 2023-11-15T06:46:26.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 msgr2=0x7f3ed8109410 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:26.931 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed8109410 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f3ec8013510 tx=0x7f3ec800b040).stop 2023-11-15T06:46:26.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 shutdown_connections 2023-11-15T06:46:26.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f3ec402ff60 0x7f3ec4032420 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:26.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 --2- 172.21.15.135:0/534971326 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f3ed80f5950 0x7f3ed8109410 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:26.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 >> 172.21.15.135:0/534971326 conn(0x7f3ed80f0260 msgr2=0x7f3ed80f0d20 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:26.932 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 shutdown_connections 2023-11-15T06:46:26.933 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:26.928+0000 7f3edf739700 1 -- 172.21.15.135:0/534971326 wait complete. 2023-11-15T06:46:26.933 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:27.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:26.315132+0000 mgr.smithi123.razjtn (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:27.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:27 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:26.927444+0000 mon.smithi123 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.135:0/534971326' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:28.328 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:28.329 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:29.467 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.464+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1653589493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 msgr2=0x7fcb480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:29.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.464+0000 7fcb4f1d4700 1 --2- 172.21.15.135:0/1653589493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb480f5830 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fcb3c009fb0 tx=0x7fcb3c009b30).stop 2023-11-15T06:46:29.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.464+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1653589493 shutdown_connections 2023-11-15T06:46:29.468 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.464+0000 7fcb4f1d4700 1 --2- 172.21.15.135:0/1653589493 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb480f5830 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:29.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.464+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1653589493 >> 172.21.15.135:0/1653589493 conn(0x7fcb480f0b70 msgr2=0x7fcb480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:29.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.464+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1653589493 shutdown_connections 2023-11-15T06:46:29.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.464+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1653589493 wait complete. 2023-11-15T06:46:29.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4f1d4700 1 Processor -- start 2023-11-15T06:46:29.469 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4f1d4700 1 -- start start 2023-11-15T06:46:29.470 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4f1d4700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb48109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:29.470 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4f1d4700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb48060240 con 0x7fcb480f5410 2023-11-15T06:46:29.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4cf70700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb48109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:29.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4cf70700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb48109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55386/0 (socket says 172.21.15.135:55386) 2023-11-15T06:46:29.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4cf70700 1 -- 172.21.15.135:0/1747444934 learned_addr learned my addr 172.21.15.135:0/1747444934 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:29.472 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.465+0000 7fcb4cf70700 1 -- 172.21.15.135:0/1747444934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c005040 con 0x7fcb480f5410 2023-11-15T06:46:29.473 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.466+0000 7fcb4cf70700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb48109430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fcb3c013510 tx=0x7fcb3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:29.473 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.466+0000 7fcb457fa700 1 -- 172.21.15.135:0/1747444934 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c009320 con 0x7fcb480f5410 2023-11-15T06:46:29.473 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.466+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb48060440 con 0x7fcb480f5410 2023-11-15T06:46:29.473 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.466+0000 7fcb457fa700 1 -- 172.21.15.135:0/1747444934 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb3c013f00 con 0x7fcb480f5410 2023-11-15T06:46:29.473 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.466+0000 7fcb457fa700 1 -- 172.21.15.135:0/1747444934 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c014d90 con 0x7fcb480f5410 2023-11-15T06:46:29.473 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.466+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb48109ae0 con 0x7fcb480f5410 2023-11-15T06:46:29.474 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.467+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb48045ad0 con 0x7fcb480f5410 2023-11-15T06:46:29.474 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.467+0000 7fcb457fa700 1 -- 172.21.15.135:0/1747444934 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb3c014420 con 0x7fcb480f5410 2023-11-15T06:46:29.475 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.467+0000 7fcb457fa700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb3802ff60 0x7fcb38032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:29.475 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.467+0000 7fcb457fa700 1 -- 172.21.15.135:0/1747444934 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb3c041050 con 0x7fcb480f5410 2023-11-15T06:46:29.475 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.467+0000 7fcb47fff700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb3802ff60 0x7fcb38032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:29.475 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.467+0000 7fcb47fff700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb3802ff60 0x7fcb38032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fcb340096a0 tx=0x7fcb34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:29.476 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.471+0000 7fcb457fa700 1 -- 172.21.15.135:0/1747444934 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb3c0095f0 con 0x7fcb480f5410 2023-11-15T06:46:29.684 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.680+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb480fae60 con 0x7fcb480f5410 2023-11-15T06:46:29.685 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.681+0000 7fcb457fa700 1 -- 172.21.15.135:0/1747444934 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb3c01db70 con 0x7fcb480f5410 2023-11-15T06:46:29.685 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:29.685 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:29.686 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.682+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb3802ff60 msgr2=0x7fcb38032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.682+0000 7fcb4f1d4700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb3802ff60 0x7fcb38032420 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fcb340096a0 tx=0x7fcb34006b40).stop 2023-11-15T06:46:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.682+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 msgr2=0x7fcb48109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.682+0000 7fcb4f1d4700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb48109430 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fcb3c013510 tx=0x7fcb3c00b040).stop 2023-11-15T06:46:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.683+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 shutdown_connections 2023-11-15T06:46:29.687 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.683+0000 7fcb4f1d4700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7fcb3802ff60 0x7fcb38032420 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:29.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.683+0000 7fcb4f1d4700 1 --2- 172.21.15.135:0/1747444934 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7fcb480f5410 0x7fcb48109430 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:29.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.683+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 >> 172.21.15.135:0/1747444934 conn(0x7fcb480f0b70 msgr2=0x7fcb480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:29.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.683+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 shutdown_connections 2023-11-15T06:46:29.688 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:29.683+0000 7fcb4f1d4700 1 -- 172.21.15.135:0/1747444934 wait complete. 2023-11-15T06:46:29.688 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:29.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:29 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:28.315546+0000 mgr.smithi123.razjtn (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:30.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:30 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:29.682204+0000 mon.smithi123 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.135:0/1747444934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:31.070 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:31.071 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:31.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:31 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:30.315847+0000 mgr.smithi123.razjtn (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:32.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.223+0000 7f63d6904700 1 -- 172.21.15.135:0/4177619305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 msgr2=0x7f63d00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:32.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.223+0000 7f63d6904700 1 --2- 172.21.15.135:0/4177619305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d00f5830 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7f63c0009fb0 tx=0x7f63c0009b30).stop 2023-11-15T06:46:32.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.223+0000 7f63d6904700 1 -- 172.21.15.135:0/4177619305 shutdown_connections 2023-11-15T06:46:32.227 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.223+0000 7f63d6904700 1 --2- 172.21.15.135:0/4177619305 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d00f5830 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:32.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.223+0000 7f63d6904700 1 -- 172.21.15.135:0/4177619305 >> 172.21.15.135:0/4177619305 conn(0x7f63d00f0b70 msgr2=0x7f63d00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:32.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.223+0000 7f63d6904700 1 -- 172.21.15.135:0/4177619305 shutdown_connections 2023-11-15T06:46:32.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.223+0000 7f63d6904700 1 -- 172.21.15.135:0/4177619305 wait complete. 2023-11-15T06:46:32.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.224+0000 7f63d6904700 1 Processor -- start 2023-11-15T06:46:32.228 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.224+0000 7f63d6904700 1 -- start start 2023-11-15T06:46:32.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.224+0000 7f63d6904700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:32.230 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.224+0000 7f63d6904700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63d0060280 con 0x7f63d00f5410 2023-11-15T06:46:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.224+0000 7f63cffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.224+0000 7f63cffff700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55404/0 (socket says 172.21.15.135:55404) 2023-11-15T06:46:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.224+0000 7f63cffff700 1 -- 172.21.15.135:0/3641917131 learned_addr learned my addr 172.21.15.135:0/3641917131 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:32.231 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.225+0000 7f63cffff700 1 -- 172.21.15.135:0/3641917131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63c0005040 con 0x7f63d00f5410 2023-11-15T06:46:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.225+0000 7f63cffff700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f63c0013510 tx=0x7f63c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.225+0000 7f63ccff9700 1 -- 172.21.15.135:0/3641917131 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63c0009320 con 0x7f63d00f5410 2023-11-15T06:46:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.225+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63d0060480 con 0x7f63d00f5410 2023-11-15T06:46:32.232 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.225+0000 7f63ccff9700 1 -- 172.21.15.135:0/3641917131 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63c0013f00 con 0x7f63d00f5410 2023-11-15T06:46:32.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.225+0000 7f63ccff9700 1 -- 172.21.15.135:0/3641917131 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63c0014d90 con 0x7f63d00f5410 2023-11-15T06:46:32.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.225+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63d0109b20 con 0x7f63d00f5410 2023-11-15T06:46:32.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.226+0000 7f63ccff9700 1 -- 172.21.15.135:0/3641917131 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63c00263a0 con 0x7f63d00f5410 2023-11-15T06:46:32.233 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.226+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63d0045ad0 con 0x7f63d00f5410 2023-11-15T06:46:32.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.226+0000 7f63ccff9700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f63b802ff60 0x7f63b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:32.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.226+0000 7f63ccff9700 1 -- 172.21.15.135:0/3641917131 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63c00412f0 con 0x7f63d00f5410 2023-11-15T06:46:32.234 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.226+0000 7f63cf7fe700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f63b802ff60 0x7f63b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:32.235 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.227+0000 7f63cf7fe700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f63b802ff60 0x7f63b8032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f63c40096a0 tx=0x7f63c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:32.236 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.230+0000 7f63ccff9700 1 -- 172.21.15.135:0/3641917131 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63c00126f0 con 0x7f63d00f5410 2023-11-15T06:46:32.439 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.436+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63d010a340 con 0x7f63d00f5410 2023-11-15T06:46:32.440 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.437+0000 7f63ccff9700 1 -- 172.21.15.135:0/3641917131 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f63c0012330 con 0x7f63d00f5410 2023-11-15T06:46:32.440 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:32.441 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:32.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f63b802ff60 msgr2=0x7f63b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:32.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f63b802ff60 0x7f63b8032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7f63c40096a0 tx=0x7f63c4006b40).stop 2023-11-15T06:46:32.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 msgr2=0x7f63d01093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:32.442 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7f63c0013510 tx=0x7f63c000b040).stop 2023-11-15T06:46:32.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 shutdown_connections 2023-11-15T06:46:32.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f63b802ff60 0x7f63b8032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:32.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 --2- 172.21.15.135:0/3641917131 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f63d00f5410 0x7f63d01093f0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:32.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 >> 172.21.15.135:0/3641917131 conn(0x7f63d00f0b70 msgr2=0x7f63d00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:32.443 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 shutdown_connections 2023-11-15T06:46:32.444 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:32.439+0000 7f63d6904700 1 -- 172.21.15.135:0/3641917131 wait complete. 2023-11-15T06:46:32.444 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:32.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:32 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:32.438311+0000 mon.smithi123 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.135:0/3641917131' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:33.843 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2023-11-15T06:46:33.844 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 -- ceph mon dump -f json 2023-11-15T06:46:33.906 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:33 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:32.316173+0000 mgr.smithi123.razjtn (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:34.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.956+0000 7f441406f700 1 -- 172.21.15.135:0/1898021887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 msgr2=0x7f440c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:34.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.956+0000 7f441406f700 1 --2- 172.21.15.135:0/1898021887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c0f5830 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f43fc009fb0 tx=0x7f43fc009b30).stop 2023-11-15T06:46:34.961 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.956+0000 7f441406f700 1 -- 172.21.15.135:0/1898021887 shutdown_connections 2023-11-15T06:46:34.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.956+0000 7f441406f700 1 --2- 172.21.15.135:0/1898021887 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c0f5830 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:34.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.956+0000 7f441406f700 1 -- 172.21.15.135:0/1898021887 >> 172.21.15.135:0/1898021887 conn(0x7f440c0f0b70 msgr2=0x7f440c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:34.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.956+0000 7f441406f700 1 -- 172.21.15.135:0/1898021887 shutdown_connections 2023-11-15T06:46:34.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.956+0000 7f441406f700 1 -- 172.21.15.135:0/1898021887 wait complete. 2023-11-15T06:46:34.962 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.957+0000 7f441406f700 1 Processor -- start 2023-11-15T06:46:34.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.957+0000 7f441406f700 1 -- start start 2023-11-15T06:46:34.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.957+0000 7f441406f700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:34.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.957+0000 7f441406f700 1 -- --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f440c060240 con 0x7f440c0f5410 2023-11-15T06:46:34.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.957+0000 7f4411e0b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:34.963 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.957+0000 7f4411e0b700 1 --2- >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.123:3300/0 says I am v2:172.21.15.135:55424/0 (socket says 172.21.15.135:55424) 2023-11-15T06:46:34.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.957+0000 7f4411e0b700 1 -- 172.21.15.135:0/2037208869 learned_addr learned my addr 172.21.15.135:0/2037208869 (peer_addr_for_me v2:172.21.15.135:0/0) 2023-11-15T06:46:34.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.958+0000 7f4411e0b700 1 -- 172.21.15.135:0/2037208869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f43fc005040 con 0x7f440c0f5410 2023-11-15T06:46:34.965 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.958+0000 7f4411e0b700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f43fc013510 tx=0x7f43fc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:34.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.958+0000 7f44027fc700 1 -- 172.21.15.135:0/2037208869 <== mon.0 v2:172.21.15.123:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43fc009320 con 0x7f440c0f5410 2023-11-15T06:46:34.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.958+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f440c060440 con 0x7f440c0f5410 2023-11-15T06:46:34.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.958+0000 7f44027fc700 1 -- 172.21.15.135:0/2037208869 <== mon.0 v2:172.21.15.123:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f43fc013f00 con 0x7f440c0f5410 2023-11-15T06:46:34.966 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.958+0000 7f44027fc700 1 -- 172.21.15.135:0/2037208869 <== mon.0 v2:172.21.15.123:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f43fc014d90 con 0x7f440c0f5410 2023-11-15T06:46:34.967 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.958+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f440c109ae0 con 0x7f440c0f5410 2023-11-15T06:46:34.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.959+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f440c045ad0 con 0x7f440c0f5410 2023-11-15T06:46:34.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.959+0000 7f44027fc700 1 -- 172.21.15.135:0/2037208869 <== mon.0 v2:172.21.15.123:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f43fc0095f0 con 0x7f440c0f5410 2023-11-15T06:46:34.968 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.959+0000 7f44027fc700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43f802ff60 0x7f43f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2023-11-15T06:46:34.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.959+0000 7f44027fc700 1 -- 172.21.15.135:0/2037208869 <== mon.0 v2:172.21.15.123:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f43fc041000 con 0x7f440c0f5410 2023-11-15T06:46:34.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.959+0000 7f441160a700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43f802ff60 0x7f43f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2023-11-15T06:46:34.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.959+0000 7f441160a700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43f802ff60 0x7f43f8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f44080096a0 tx=0x7f4408006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2023-11-15T06:46:34.969 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:34.963+0000 7f44027fc700 1 -- 172.21.15.135:0/2037208869 <== mon.0 v2:172.21.15.123:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f43fc014420 con 0x7f440c0f5410 2023-11-15T06:46:35.171 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.169+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 --> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f440c0fae60 con 0x7f440c0f5410 2023-11-15T06:46:35.172 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.169+0000 7f44027fc700 1 -- 172.21.15.135:0/2037208869 <== mon.0 v2:172.21.15.123:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f43fc010230 con 0x7f440c0f5410 2023-11-15T06:46:35.173 INFO:teuthology.orchestra.run.smithi135.stdout: 2023-11-15T06:46:35.173 INFO:teuthology.orchestra.run.smithi135.stdout:{"epoch":1,"fsid":"4a6dbac0-8381-11ee-8dbe-212e2dc638e7","modified":"2023-11-15T06:36:47.245818Z","created":"2023-11-15T06:36:47.245818Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi123","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.123:3300","nonce":0},{"type":"v1","addr":"172.21.15.123:6789","nonce":0}]},"addr":"172.21.15.123:6789/0","public_addr":"172.21.15.123:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2023-11-15T06:46:35.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43f802ff60 msgr2=0x7f43f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:35.174 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43f802ff60 0x7f43f8032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f44080096a0 tx=0x7f4408006b40).stop 2023-11-15T06:46:35.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 msgr2=0x7f440c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2023-11-15T06:46:35.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c109430 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f43fc013510 tx=0x7f43fc00b040).stop 2023-11-15T06:46:35.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 shutdown_connections 2023-11-15T06:46:35.175 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:6800/1639427586,v1:172.21.15.123:6801/1639427586] conn(0x7f43f802ff60 0x7f43f8032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:35.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 --2- 172.21.15.135:0/2037208869 >> [v2:172.21.15.123:3300/0,v1:172.21.15.123:6789/0] conn(0x7f440c0f5410 0x7f440c109430 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2023-11-15T06:46:35.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 >> 172.21.15.135:0/2037208869 conn(0x7f440c0f0b70 msgr2=0x7f440c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2023-11-15T06:46:35.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 shutdown_connections 2023-11-15T06:46:35.176 INFO:teuthology.orchestra.run.smithi135.stderr:2023-11-15T06:46:35.171+0000 7f441406f700 1 -- 172.21.15.135:0/2037208869 wait complete. 2023-11-15T06:46:35.176 INFO:teuthology.orchestra.run.smithi135.stderr:dumped monmap epoch 1 2023-11-15T06:46:35.595 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2023-11-15T06:46:35.595 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2023-11-15T06:46:35.596 DEBUG:teuthology.orchestra.run.smithi123:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:46:35.622 DEBUG:teuthology.orchestra.run.smithi135:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2023-11-15T06:46:35.642 INFO:tasks.cephadm:Stopping all daemons... 2023-11-15T06:46:35.643 INFO:tasks.cephadm.mon.smithi123:Stopping mon.smithi123... 2023-11-15T06:46:35.643 DEBUG:teuthology.orchestra.run.smithi123:> sudo systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123 2023-11-15T06:46:35.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: cluster 2023-11-15T06:46:34.316504+0000 mgr.smithi123.razjtn (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2023-11-15T06:46:35.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: audit 2023-11-15T06:46:35.170439+0000 mon.smithi123 (mon.0) 325 : audit [DBG] from='client.? 172.21.15.135:0/2037208869' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2023-11-15T06:46:35.907 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:35 smithi123 systemd[1]: Stopping Ceph mon.smithi123 for 4a6dbac0-8381-11ee-8dbe-212e2dc638e7... 2023-11-15T06:46:35.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:46:35.783+0000 7fd85003f700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi123 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2023-11-15T06:46:35.908 INFO:journalctl@ceph.mon.smithi123.smithi123.stdout:Nov 15 06:46:35 smithi123 ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7-mon.smithi123[112306]: debug 2023-11-15T06:46:35.783+0000 7fd85003f700 -1 mon.smithi123@0(leader) e1 *** Got Signal Terminated *** 2023-11-15T06:46:36.167 DEBUG:teuthology.orchestra.run.smithi123:> sudo pkill -f 'journalctl -f -n 0 -u ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi123.service' 2023-11-15T06:46:36.215 DEBUG:teuthology.orchestra.run:got remote process result: None 2023-11-15T06:46:36.215 INFO:tasks.cephadm.mon.smithi123:Stopped mon.smithi123 2023-11-15T06:46:36.215 INFO:tasks.cephadm.mon.smithi135:Stopping mon.smithi135... 2023-11-15T06:46:36.216 DEBUG:teuthology.orchestra.run.smithi135:> sudo systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135 2023-11-15T06:46:36.248 INFO:teuthology.orchestra.run.smithi135.stderr:Failed to stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135.service: Unit ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135.service not loaded. 2023-11-15T06:46:36.249 DEBUG:teuthology.orchestra.run:got remote process result: 5 2023-11-15T06:46:36.250 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi135" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi135 with status 5: 'sudo systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135' 2023-11-15T06:46:36.251 INFO:tasks.cephadm:Archiving crash dumps... 2023-11-15T06:46:36.253 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622/remote/ubuntu@smithi123.front.sepia.ceph.com/crash 2023-11-15T06:46:36.255 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T06:46:36.295 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/crash to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622/remote/ubuntu@smithi135.front.sepia.ceph.com/crash 2023-11-15T06:46:36.296 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/crash -- . 2023-11-15T06:46:36.343 INFO:teuthology.orchestra.run.smithi135.stderr:tar: /var/lib/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/crash: Cannot open: No such file or directory 2023-11-15T06:46:36.343 INFO:teuthology.orchestra.run.smithi135.stderr:tar: Error is not recoverable: exiting now 2023-11-15T06:46:36.344 INFO:tasks.cephadm:Checking cluster log for badness... 2023-11-15T06:46:36.345 DEBUG:teuthology.orchestra.run.smithi123:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2023-11-15T06:46:36.375 INFO:teuthology.orchestra.run.smithi123.stderr:grep: /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph.log: No such file or directory 2023-11-15T06:46:36.377 INFO:tasks.cephadm:Compressing logs... 2023-11-15T06:46:36.378 DEBUG:teuthology.orchestra.run.smithi123:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T06:46:36.420 DEBUG:teuthology.orchestra.run.smithi135:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2023-11-15T06:46:36.445 INFO:teuthology.orchestra.run.smithi123.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T06:46:36.445 INFO:teuthology.orchestra.run.smithi123.stderr:: No such file or directory 2023-11-15T06:46:36.445 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-volume.log 2023-11-15T06:46:36.446 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-mon.smithi123.log 2023-11-15T06:46:36.446 INFO:teuthology.orchestra.run.smithi123.stderr:gzip -5 --verbose -- /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi123.razjtn.log 2023-11-15T06:46:36.447 INFO:teuthology.orchestra.run.smithi135.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2023-11-15T06:46:36.447 INFO:teuthology.orchestra.run.smithi135.stderr:: No such file or directory 2023-11-15T06:46:36.447 INFO:teuthology.orchestra.run.smithi123.stderr:/var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-volume.log: /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-mon.smithi123.log: /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi123.razjtn.log: 90.2% -- replaced with /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-volume.log.gz 2023-11-15T06:46:36.448 INFO:teuthology.orchestra.run.smithi135.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T06:46:36.449 INFO:teuthology.orchestra.run.smithi135.stderr: 2023-11-15T06:46:36.450 INFO:teuthology.orchestra.run.smithi135.stderr:real 0m0.013s 2023-11-15T06:46:36.450 INFO:teuthology.orchestra.run.smithi135.stderr:user 0m0.011s 2023-11-15T06:46:36.450 INFO:teuthology.orchestra.run.smithi135.stderr:sys 0m0.010s 2023-11-15T06:46:36.451 INFO:teuthology.orchestra.run.smithi123.stderr: 92.4% -- replaced with /var/log/ceph/cephadm.log.gz 2023-11-15T06:46:36.520 INFO:teuthology.orchestra.run.smithi123.stderr:gzip: /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi123.razjtn.log: file size changed while zipping 2023-11-15T06:46:36.520 INFO:teuthology.orchestra.run.smithi123.stderr: 92.6% -- replaced with /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-mgr.smithi123.razjtn.log.gz 2023-11-15T06:46:36.585 INFO:teuthology.orchestra.run.smithi123.stderr: 90.5% -- replaced with /var/log/ceph/4a6dbac0-8381-11ee-8dbe-212e2dc638e7/ceph-mon.smithi123.log.gz 2023-11-15T06:46:36.586 INFO:teuthology.orchestra.run.smithi123.stderr: 2023-11-15T06:46:36.587 INFO:teuthology.orchestra.run.smithi123.stderr:real 0m0.152s 2023-11-15T06:46:36.587 INFO:teuthology.orchestra.run.smithi123.stderr:user 0m0.227s 2023-11-15T06:46:36.587 INFO:teuthology.orchestra.run.smithi123.stderr:sys 0m0.015s 2023-11-15T06:46:36.587 INFO:tasks.cephadm:Archiving logs... 2023-11-15T06:46:36.588 DEBUG:teuthology.misc:Transferring archived files from smithi123:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622/remote/ubuntu@smithi123.front.sepia.ceph.com/log 2023-11-15T06:46:36.589 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T06:46:36.704 DEBUG:teuthology.misc:Transferring archived files from smithi135:/var/log/ceph to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622/remote/ubuntu@smithi135.front.sepia.ceph.com/log 2023-11-15T06:46:36.705 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /var/log/ceph -- . 2023-11-15T06:46:36.737 INFO:tasks.cephadm:Removing cluster... 2023-11-15T06:46:36.738 DEBUG:teuthology.orchestra.run.smithi123:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 --force 2023-11-15T06:48:06.046 DEBUG:teuthology.orchestra.run.smithi135:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 4a6dbac0-8381-11ee-8dbe-212e2dc638e7 --force 2023-11-15T06:48:06.354 INFO:tasks.cephadm:Removing cephadm ... 2023-11-15T06:48:06.355 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T06:48:06.374 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf /home/ubuntu/cephtest/cephadm 2023-11-15T06:48:06.393 INFO:tasks.cephadm:Teardown complete 2023-11-15T06:48:06.394 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi135 with status 5: 'sudo systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135' 2023-11-15T06:48:06.499 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=3e347ef7e449411f90cc94a0ae3ce330 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_620f25c1794f62207424fc9319427943fb8d1fae/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 96, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/remote.py", line 522, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_6899cd26fceddb2fec83dc1a1349394b28c8998e/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi135 with status 5: 'sudo systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135' 2023-11-15T06:48:06.504 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2023-11-15T06:48:06.518 DEBUG:teuthology.run_tasks:Unwinding manager install 2023-11-15T06:48:06.528 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2023-11-15T06:48:06.528 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T06:48:06.553 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2023-11-15T06:48:06.635 DEBUG:teuthology.run_tasks:Unwinding manager clock 2023-11-15T06:48:06.644 INFO:teuthology.task.clock:Checking final clock skew... 2023-11-15T06:48:06.645 DEBUG:teuthology.orchestra.run.smithi123:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:48:06.648 DEBUG:teuthology.orchestra.run.smithi135:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2023-11-15T06:48:06.662 INFO:teuthology.orchestra.run.smithi123.stderr:bash: ntpq: command not found 2023-11-15T06:48:06.664 INFO:teuthology.orchestra.run.smithi135.stderr:bash: ntpq: command not found 2023-11-15T06:48:06.666 INFO:teuthology.orchestra.run.smithi123.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:48:06.667 INFO:teuthology.orchestra.run.smithi123.stdout:=============================================================================== 2023-11-15T06:48:06.667 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +1912us[+1910us] +/- 89ms 2023-11-15T06:48:06.667 INFO:teuthology.orchestra.run.smithi123.stdout:^* hv02.front.sepia.ceph.com 3 6 377 17 -942us[ -944us] +/- 62ms 2023-11-15T06:48:06.668 INFO:teuthology.orchestra.run.smithi123.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 19 -374us[ -377us] +/- 67ms 2023-11-15T06:48:06.668 INFO:teuthology.orchestra.run.smithi123.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:48:06.669 INFO:teuthology.orchestra.run.smithi135.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2023-11-15T06:48:06.670 INFO:teuthology.orchestra.run.smithi135.stdout:=============================================================================== 2023-11-15T06:48:06.670 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 17 +1911us[+1922us] +/- 89ms 2023-11-15T06:48:06.670 INFO:teuthology.orchestra.run.smithi135.stdout:^* hv02.front.sepia.ceph.com 3 6 377 14 -924us[ -914us] +/- 62ms 2023-11-15T06:48:06.671 INFO:teuthology.orchestra.run.smithi135.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 16 -398us[ -387us] +/- 67ms 2023-11-15T06:48:06.671 INFO:teuthology.orchestra.run.smithi135.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2023-11-15T06:48:06.671 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2023-11-15T06:48:06.683 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2023-11-15T06:48:06.684 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2023-11-15T06:48:06.696 DEBUG:teuthology.orchestra.run.smithi123:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T06:48:06.797 DEBUG:teuthology.orchestra.run.smithi135:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2023-11-15T06:48:06.887 DEBUG:teuthology.orchestra.run.smithi123:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:48:06.919 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:48:06.919 DEBUG:teuthology.orchestra.run.smithi135:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2023-11-15T06:48:06.951 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:48:06.952 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2023-11-15T06:48:06.963 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2023-11-15T06:48:06.973 INFO:teuthology.task.internal:Duration was 1382.796690 seconds 2023-11-15T06:48:06.974 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2023-11-15T06:48:06.985 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2023-11-15T06:48:06.986 DEBUG:teuthology.orchestra.run.smithi123:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T06:48:06.988 DEBUG:teuthology.orchestra.run.smithi135:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2023-11-15T06:48:07.032 INFO:teuthology.orchestra.run.smithi123.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:48:07.038 INFO:teuthology.orchestra.run.smithi135.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2023-11-15T06:48:07.417 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2023-11-15T06:48:07.418 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi123.front.sepia.ceph.com 2023-11-15T06:48:07.418 DEBUG:teuthology.orchestra.run.smithi123:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T06:48:07.441 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi135.front.sepia.ceph.com 2023-11-15T06:48:07.442 DEBUG:teuthology.orchestra.run.smithi135:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2023-11-15T06:48:07.467 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2023-11-15T06:48:07.467 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T06:48:07.484 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2023-11-15T06:48:07.551 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2023-11-15T06:48:07.552 DEBUG:teuthology.orchestra.run.smithi123:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T06:48:07.572 DEBUG:teuthology.orchestra.run.smithi135:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2023-11-15T06:48:07.676 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2023-11-15T06:48:07.696 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2023-11-15T06:48:07.697 DEBUG:teuthology.orchestra.run.smithi123:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T06:48:07.724 DEBUG:teuthology.orchestra.run.smithi135:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2023-11-15T06:48:07.752 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2023-11-15T06:48:07.763 DEBUG:teuthology.orchestra.run.smithi123:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:48:07.767 DEBUG:teuthology.orchestra.run.smithi135:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:48:07.791 INFO:teuthology.orchestra.run.smithi123.stdout:kernel.core_pattern = core 2023-11-15T06:48:07.821 INFO:teuthology.orchestra.run.smithi135.stdout:kernel.core_pattern = core 2023-11-15T06:48:07.848 DEBUG:teuthology.orchestra.run.smithi123:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:48:07.879 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:48:07.880 DEBUG:teuthology.orchestra.run.smithi135:> test -e /home/ubuntu/cephtest/archive/coredump 2023-11-15T06:48:07.904 DEBUG:teuthology.orchestra.run:got remote process result: 1 2023-11-15T06:48:07.905 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2023-11-15T06:48:07.916 INFO:teuthology.task.internal:Transferring archived files... 2023-11-15T06:48:07.918 DEBUG:teuthology.misc:Transferring archived files from smithi123:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622/remote/smithi123 2023-11-15T06:48:07.920 DEBUG:teuthology.orchestra.run.smithi123:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T06:48:07.959 DEBUG:teuthology.misc:Transferring archived files from smithi135:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622/remote/smithi135 2023-11-15T06:48:07.961 DEBUG:teuthology.orchestra.run.smithi135:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2023-11-15T06:48:08.001 INFO:teuthology.task.internal:Removing archive directory... 2023-11-15T06:48:08.002 DEBUG:teuthology.orchestra.run.smithi123:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T06:48:08.005 DEBUG:teuthology.orchestra.run.smithi135:> rm -rf -- /home/ubuntu/cephtest/archive 2023-11-15T06:48:08.053 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2023-11-15T06:48:08.065 INFO:teuthology.task.internal:Not uploading archives. 2023-11-15T06:48:08.066 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2023-11-15T06:48:08.080 INFO:teuthology.task.internal:Tidying up after the test... 2023-11-15T06:48:08.081 DEBUG:teuthology.orchestra.run.smithi123:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T06:48:08.084 DEBUG:teuthology.orchestra.run.smithi135:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2023-11-15T06:48:08.099 INFO:teuthology.orchestra.run.smithi123.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 06:48 /home/ubuntu/cephtest 2023-11-15T06:48:08.111 INFO:teuthology.orchestra.run.smithi135.stdout: 525357 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Nov 15 06:48 /home/ubuntu/cephtest 2023-11-15T06:48:08.112 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2023-11-15T06:48:08.124 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2023-11-15T06:48:08.150 INFO:teuthology.nuke:Checking targets against current locks 2023-11-15T06:48:08.178 DEBUG:teuthology.nuke:shortname: smithi123 2023-11-15T06:48:08.179 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:48:08.204 DEBUG:teuthology.nuke:shortname: smithi135 2023-11-15T06:48:08.205 INFO:teuthology.task.internal.check_lock:Checking locks... 2023-11-15T06:48:08.210 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi123.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:16:39.760724', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBArzQg8bBU/6MM3lFQxUyLexbvAq4wWJumoQgZ4oJUVF4GWiBYIEpFJtqLXsgo4JfWOst+BCXkyuhGOolf+pGks='} 2023-11-15T06:48:08.237 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi135.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2023-11-14_20:45:49-rados-wip-yuri4-testing-2023-11-13-0820-pacific-distro-default-smithi/7458622', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2023-11-15 06:16:39.762082', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBJQq44W7J6KrofH3PchCIcL9iFw+bUWcuxZtbLws7vGqJip8dd4tXegIuIZuE2K8bdDTUA8GzDYV1NHrSIP45QA='} 2023-11-15T06:48:08.277 INFO:teuthology.orchestra.console.smithi123:Power off 2023-11-15T06:48:08.278 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T06:48:08.306 DEBUG:teuthology.orchestra.console.smithi123:power off output: Chassis Power Control: Down/Off 2023-11-15T06:48:08.307 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:48:08.328 INFO:teuthology.orchestra.console.smithi135:Power off 2023-11-15T06:48:08.329 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2023-11-15T06:48:08.342 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-11-15T06:48:08.358 DEBUG:teuthology.orchestra.console.smithi135:power off output: Chassis Power Control: Down/Off 2023-11-15T06:48:08.359 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:48:08.398 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2023-11-15T06:48:12.345 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:48:12.400 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:48:12.471 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is on 2023-11-15T06:48:12.515 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is on 2023-11-15T06:48:16.474 DEBUG:teuthology.orchestra.console.smithi123:pexpect command: ipmitool -H smithi123.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:48:16.516 DEBUG:teuthology.orchestra.console.smithi135:pexpect command: ipmitool -H smithi135.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2023-11-15T06:48:16.595 DEBUG:teuthology.orchestra.console.smithi123:check power output: Chassis Power is off 2023-11-15T06:48:16.635 DEBUG:teuthology.orchestra.console.smithi135:check power output: Chassis Power is off 2023-11-15T06:48:16.698 INFO:teuthology.orchestra.console.smithi123:Power off completed 2023-11-15T06:48:16.736 INFO:teuthology.orchestra.console.smithi135:Power off completed 2023-11-15T06:48:16.841 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/2 2-allow_standby_replay/no 3-inline/yes 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1382.796689748764 failure_reason: 'Command failed on smithi135 with status 5: ''sudo systemctl stop ceph-4a6dbac0-8381-11ee-8dbe-212e2dc638e7@mon.smithi135''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=3e347ef7e449411f90cc94a0ae3ce330 status: fail success: false 2023-11-15T06:48:16.841 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2023-11-15T06:48:16.940 INFO:teuthology.run:FAIL