2024-01-27T23:11:18.740 INFO:root:teuthology version: 0.0.1.dev251+gc856e0f 2024-01-27T23:11:18.740 DEBUG:teuthology.run:Teuthology command: teuthology --verbose --owner scheduled_yuriw@teuthology --description rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} --name yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi --archive /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122 -- /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122/orig.config.yaml 2024-01-27T23:11:18.788 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-27T23:11:18.872 INFO:teuthology.run:Config: archive_path: /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122 branch: pacific-release description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} email: null first_in_suite: false job_id: '7535122' kernel: kdb: true sha1: distro last_in_suite: false machine_type: smithi meta: - desc: 'setup ceph/pacific v16.2.4 ' name: yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi no_nested_subset: false nuke-on-error: true os_type: centos os_version: 8.stream overrides: admin_socket: branch: pacific-release ceph: conf: client: client mount timeout: 600 debug client: 20 debug ms: 1 rados mon op timeout: 900 rados osd op timeout: 900 global: mon pg warn min per osd: 0 mds: debug mds: 20 debug mds balancer: 20 debug ms: 1 mds debug frag: true mds debug scatterstat: true mds op complaint time: 180 mds verify scatter: true osd op complaint time: 180 rados mon op timeout: 900 rados osd op timeout: 900 mgr: debug mgr: 20 debug ms: 1 mon: debug mon: 20 debug ms: 1 debug paxos: 20 mon op complaint time: 120 osd: bdev async discard: true bdev enable discard: true bluestore allocator: bitmap bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug ms: 1 debug osd: 20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore osd op complaint time: 180 flavor: default fs: xfs log-ignorelist: - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - overall HEALTH_ - \(FS_DEGRADED\) - \(MDS_FAILED\) - \(MDS_DEGRADED\) - \(FS_WITH_FAILED_MDS\) - \(MDS_DAMAGE\) - \(MDS_ALL_DOWN\) - \(MDS_UP_LESS_THAN_MAX\) - \(FS_INLINE_DATA_DEPRECATED\) - overall HEALTH_ - \(OSD_DOWN\) - \(OSD_ - but it is still running - is not responding sha1: a72a5dace2353a54147d37bd2699a2a482507966 ceph-deploy: bluestore: true conf: client: log file: /var/log/ceph/ceph-$name.$pid.log mon: {} osd: bdev async discard: true bdev enable discard: true bluestore block size: 96636764160 bluestore fsck on mount: true debug bluefs: 1/20 debug bluestore: 1/20 debug rocksdb: 4/10 mon osd backfillfull_ratio: 0.85 mon osd full ratio: 0.9 mon osd nearfull ratio: 0.8 osd failsafe full ratio: 0.95 osd objectstore: bluestore fs: xfs install: ceph: flavor: default sha1: a72a5dace2353a54147d37bd2699a2a482507966 selinux: whitelist: - scontext=system_u:system_r:logrotate_t:s0 thrashosds: bdev_inject_crash: 2 bdev_inject_crash_probability: 0.5 workunit: branch: pacific-release sha1: 585bdf827237450fb80df4a856eba434a64948fa owner: scheduled_yuriw@teuthology priority: 75 repo: https://github.com/ceph/ceph-ci.git roles: - - host.a - client.0 - osd.0 - osd.1 - osd.2 - - host.b - client.1 - osd.3 - osd.4 - osd.5 seed: 6332 sha1: a72a5dace2353a54147d37bd2699a2a482507966 sleep_before_teardown: 0 subset: 111/120000 suite: rados suite_branch: pacific-release suite_path: /home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa suite_relpath: qa suite_repo: https://github.com/ceph/ceph-ci.git suite_sha1: 585bdf827237450fb80df4a856eba434a64948fa targets: smithi121.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHWGgLFI/ZvCOpCwT7jcXXfQi77YaJe7dKPTApJcsX4KnN/15O0Y+/1pOo+BQ/PWu/WwXla2EVZGsEG6QKm3+S4= smithi194.front.sepia.ceph.com: ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM8Nzw7Mdc6JkoonhBu9o8XjQxMVwogjIQPozN+3mETOrwYHuQVrKdGqNGS/m9MhFAmdsIFmuURupmmSkE6ENT8= tasks: - pexec: all: - sudo cp /etc/containers/registries.conf /etc/containers/registries.conf.backup - sudo dnf -y module reset container-tools - sudo dnf -y module install container-tools --allowerasing --nobest - sudo cp /etc/containers/registries.conf.backup /etc/containers/registries.conf - pexec: clients: - sudo modprobe -r ceph - sudo modprobe ceph disable_send_metrics=on - install: exclude_packages: - ceph-volume tag: v16.2.4 - print: '**** done install task...' - cephadm: avoid_pacific_features: true cephadm_branch: v16.2.4 cephadm_git_url: https://github.com/ceph/ceph image: docker.io/ceph/ceph:v16.2.4 roleless: true - print: '**** done starting v16.2.4' - cephadm.shell: host.a: - ceph orch status - ceph orch ps - ceph orch ls - ceph orch host ls - ceph orch device ls - cephadm.shell: host.a: - ceph fs volume create cephfs --placement=4 - ceph fs dump - cephadm.shell: host.a: - ceph fs set cephfs max_mds 1 - cephadm.shell: host.a: - ceph fs set cephfs allow_standby_replay true - cephadm.shell: host.a: - ceph fs set cephfs inline_data false - cephadm.shell: host.a: - ceph fs dump - ceph --format=json fs dump | jq -e ".filesystems | length == 1" - while ! ceph --format=json mds versions | jq -e ". | add == 4"; do sleep 1; done - fs.pre_upgrade_save: null - kclient: null - print: '**** done client' - parallel: - upgrade-tasks - workload-tasks - cephadm.shell: host.a: - ceph fs dump - fs.post_upgrade_checks: null teuthology: fragments_dropped: [] meta: {} postmerge: [] teuthology_branch: main teuthology_sha1: c856e0f9244dd29a50697123b480d63f16a4ed81 timestamp: 2024-01-26_16:10:31 tube: smithi upgrade-tasks: sequential: - cephadm.shell: env: - sha1 host.a: - ceph config set mon mon_warn_on_insecure_global_id_reclaim false --force - ceph config set mon mon_warn_on_insecure_global_id_reclaim_allowed false --force - ceph config set global log_to_journald false --force - ceph orch upgrade start --image quay.ceph.io/ceph-ci/ceph:$sha1 - cephadm.shell: env: - sha1 host.a: - while ceph orch upgrade status | jq '.in_progress' | grep true && ! ceph orch upgrade status | jq '.message' | grep Error ; do ceph orch ps ; ceph versions ; ceph fs dump; ceph orch upgrade status ; sleep 30 ; done - ceph orch ps - ceph versions - echo "wait for servicemap items w/ changing names to refresh" - sleep 60 - ceph orch ps - ceph versions - ceph versions | jq -e '.overall | length == 1' - ceph versions | jq -e '.overall | keys' | grep $sha1 user: yuriw verbose: true worker_log: /home/teuthworker/archive/worker_logs/dispatcher.smithi.2226885 workload-tasks: sequential: - workunit: clients: all: - suites/fsstress.sh 2024-01-27T23:11:18.872 INFO:teuthology.run:suite_path is set to /home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa; will attempt to use it 2024-01-27T23:11:18.873 INFO:teuthology.run:Found tasks at /home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks 2024-01-27T23:11:18.873 INFO:teuthology.run_tasks:Running task internal.check_packages... 2024-01-27T23:11:18.875 INFO:teuthology.task.internal:Checking packages... 2024-01-27T23:11:18.898 INFO:teuthology.task.internal:Checking packages for os_type 'centos', flavor 'default' and ceph hash 'a72a5dace2353a54147d37bd2699a2a482507966' 2024-01-27T23:11:18.899 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using branch 2024-01-27T23:11:18.899 INFO:teuthology.packaging:ref: None 2024-01-27T23:11:18.899 INFO:teuthology.packaging:tag: None 2024-01-27T23:11:18.899 INFO:teuthology.packaging:branch: pacific-release 2024-01-27T23:11:18.899 INFO:teuthology.packaging:sha1: a72a5dace2353a54147d37bd2699a2a482507966 2024-01-27T23:11:18.899 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&ref=pacific-release 2024-01-27T23:11:19.112 INFO:teuthology.task.internal:Found packages for ceph version 16.2.14-508.ga72a5dac 2024-01-27T23:11:19.113 INFO:teuthology.run_tasks:Running task internal.buildpackages_prep... 2024-01-27T23:11:19.121 INFO:teuthology.task.internal:no buildpackages task found 2024-01-27T23:11:19.121 INFO:teuthology.run_tasks:Running task internal.save_config... 2024-01-27T23:11:19.127 INFO:teuthology.task.internal:Saving configuration 2024-01-27T23:11:19.145 INFO:teuthology.run_tasks:Running task internal.check_lock... 2024-01-27T23:11:19.151 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T23:11:19.174 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 23:06:26.016339', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHWGgLFI/ZvCOpCwT7jcXXfQi77YaJe7dKPTApJcsX4KnN/15O0Y+/1pOo+BQ/PWu/WwXla2EVZGsEG6QKm3+S4='} 2024-01-27T23:11:19.192 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 23:06:26.017527', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM8Nzw7Mdc6JkoonhBu9o8XjQxMVwogjIQPozN+3mETOrwYHuQVrKdGqNGS/m9MhFAmdsIFmuURupmmSkE6ENT8='} 2024-01-27T23:11:19.192 INFO:teuthology.run_tasks:Running task internal.add_remotes... 2024-01-27T23:11:19.198 INFO:teuthology.task.internal:roles: ubuntu@smithi121.front.sepia.ceph.com - ['host.a', 'client.0', 'osd.0', 'osd.1', 'osd.2'] 2024-01-27T23:11:19.198 INFO:teuthology.task.internal:roles: ubuntu@smithi194.front.sepia.ceph.com - ['host.b', 'client.1', 'osd.3', 'osd.4', 'osd.5'] 2024-01-27T23:11:19.198 INFO:teuthology.run_tasks:Running task console_log... 2024-01-27T23:11:19.264 DEBUG:teuthology.exit:Installing handler: Handler(exiter=, func=.kill_console_loggers at 0x7f9f25a160d0>, signals=[15]) 2024-01-27T23:11:19.264 INFO:teuthology.run_tasks:Running task internal.connect... 2024-01-27T23:11:19.270 INFO:teuthology.task.internal:Opening connections... 2024-01-27T23:11:19.270 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi121.front.sepia.ceph.com 2024-01-27T23:11:19.272 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:11:19.346 DEBUG:teuthology.task.internal:connecting to ubuntu@smithi194.front.sepia.ceph.com 2024-01-27T23:11:19.347 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:11:19.422 INFO:teuthology.run_tasks:Running task internal.push_inventory... 2024-01-27T23:11:19.429 DEBUG:teuthology.orchestra.run.smithi121:> uname -m 2024-01-27T23:11:19.452 INFO:teuthology.orchestra.run.smithi121.stdout:x86_64 2024-01-27T23:11:19.452 DEBUG:teuthology.orchestra.run.smithi121:> cat /etc/os-release 2024-01-27T23:11:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:NAME="CentOS Stream" 2024-01-27T23:11:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION="8" 2024-01-27T23:11:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:ID="centos" 2024-01-27T23:11:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:ID_LIKE="rhel fedora" 2024-01-27T23:11:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:VERSION_ID="8" 2024-01-27T23:11:19.512 INFO:teuthology.orchestra.run.smithi121.stdout:PLATFORM_ID="platform:el8" 2024-01-27T23:11:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-27T23:11:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:ANSI_COLOR="0;31" 2024-01-27T23:11:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-27T23:11:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:HOME_URL="https://centos.org/" 2024-01-27T23:11:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-27T23:11:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-27T23:11:19.513 INFO:teuthology.orchestra.run.smithi121.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-27T23:11:19.513 INFO:teuthology.lock.ops:Updating smithi121.front.sepia.ceph.com on lock server 2024-01-27T23:11:19.536 DEBUG:teuthology.orchestra.run.smithi194:> uname -m 2024-01-27T23:11:19.558 INFO:teuthology.orchestra.run.smithi194.stdout:x86_64 2024-01-27T23:11:19.559 DEBUG:teuthology.orchestra.run.smithi194:> cat /etc/os-release 2024-01-27T23:11:19.619 INFO:teuthology.orchestra.run.smithi194.stdout:NAME="CentOS Stream" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION="8" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:ID="centos" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:ID_LIKE="rhel fedora" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:VERSION_ID="8" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:PLATFORM_ID="platform:el8" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:PRETTY_NAME="CentOS Stream 8" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:ANSI_COLOR="0;31" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:CPE_NAME="cpe:/o:centos:centos:8" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:HOME_URL="https://centos.org/" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:BUG_REPORT_URL="https://bugzilla.redhat.com/" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8" 2024-01-27T23:11:19.620 INFO:teuthology.orchestra.run.smithi194.stdout:REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream" 2024-01-27T23:11:19.621 INFO:teuthology.lock.ops:Updating smithi194.front.sepia.ceph.com on lock server 2024-01-27T23:11:19.645 INFO:teuthology.run_tasks:Running task internal.serialize_remote_roles... 2024-01-27T23:11:19.654 INFO:teuthology.run_tasks:Running task internal.check_conflict... 2024-01-27T23:11:19.660 INFO:teuthology.task.internal:Checking for old test directory... 2024-01-27T23:11:19.660 DEBUG:teuthology.orchestra.run.smithi121:> test '!' -e /home/ubuntu/cephtest 2024-01-27T23:11:19.663 DEBUG:teuthology.orchestra.run.smithi194:> test '!' -e /home/ubuntu/cephtest 2024-01-27T23:11:19.682 INFO:teuthology.run_tasks:Running task internal.check_ceph_data... 2024-01-27T23:11:19.689 INFO:teuthology.task.internal:Checking for non-empty /var/lib/ceph... 2024-01-27T23:11:19.689 DEBUG:teuthology.orchestra.run.smithi121:> test -z $(ls -A /var/lib/ceph) 2024-01-27T23:11:19.725 DEBUG:teuthology.orchestra.run.smithi194:> test -z $(ls -A /var/lib/ceph) 2024-01-27T23:11:19.771 INFO:teuthology.run_tasks:Running task internal.vm_setup... 2024-01-27T23:11:19.824 INFO:teuthology.run_tasks:Running task kernel... 2024-01-27T23:11:19.838 INFO:teuthology.task.kernel:normalize config orig: {'kdb': True, 'sha1': 'distro'} 2024-01-27T23:11:19.839 INFO:teuthology.task.kernel:config {'host.a': {'kdb': True, 'sha1': 'distro'}, 'host.b': {'kdb': True, 'sha1': 'distro'}}, timeout 300 2024-01-27T23:11:19.839 DEBUG:teuthology.orchestra.run.smithi121:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-27T23:11:19.839 DEBUG:teuthology.orchestra.run.smithi194:> test -f /run/.containerenv -o -f /.dockerenv 2024-01-27T23:11:19.858 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:11:19.859 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-01-27T23:11:19.859 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:11:19.860 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-01-27T23:11:19.918 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-489.el8.x86_64 2024-01-27T23:11:19.918 INFO:teuthology.task.kernel:Running kernel on smithi194: 4.18.0-489.el8.x86_64 2024-01-27T23:11:19.918 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum install -y kernel 2024-01-27T23:11:19.920 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-489.el8.x86_64 2024-01-27T23:11:19.921 INFO:teuthology.task.kernel:Running kernel on smithi121: 4.18.0-489.el8.x86_64 2024-01-27T23:11:19.921 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum install -y kernel 2024-01-27T23:11:23.856 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 19 MB/s | 27 MB 00:01 2024-01-27T23:11:23.986 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 18 MB/s | 27 MB 00:01 2024-01-27T23:11:32.355 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 2.8 MB/s | 10 MB 00:03 2024-01-27T23:11:32.902 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 2.6 MB/s | 10 MB 00:03 2024-01-27T23:11:34.589 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 29 kB/s | 18 kB 00:00 2024-01-27T23:11:34.928 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 43 kB/s | 18 kB 00:00 2024-01-27T23:11:35.081 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-27T23:11:35.261 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 31 kB/s | 6.9 kB 00:00 2024-01-27T23:11:35.299 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 27 kB/s | 3.5 kB 00:00 2024-01-27T23:11:35.480 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 28 kB/s | 3.5 kB 00:00 2024-01-27T23:11:36.214 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 25 MB/s | 16 MB 00:00 2024-01-27T23:11:36.951 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 10 MB/s | 16 MB 00:01 2024-01-27T23:11:39.947 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 410 kB/s | 24 kB 00:00 2024-01-27T23:11:40.650 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 554 kB/s | 24 kB 00:00 2024-01-27T23:11:42.902 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-27T23:11:42.902 INFO:teuthology.orchestra.run.smithi194.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-27T23:11:43.000 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:11:43.000 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout:Install 3 Packages 2024-01-27T23:11:43.001 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:11:43.002 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 90 M 2024-01-27T23:11:43.002 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 96 M 2024-01-27T23:11:43.002 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:11:43.554 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-348.el8.x86_64 is already installed. 2024-01-27T23:11:43.555 INFO:teuthology.orchestra.run.smithi121.stdout:Package kernel-4.18.0-489.el8.x86_64 is already installed. 2024-01-27T23:11:43.679 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:11:43.679 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:11:43.679 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:11:43.679 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: kernel x86_64 4.18.0-536.el8 baseos 10 M 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-core x86_64 4.18.0-536.el8 baseos 44 M 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules x86_64 4.18.0-536.el8 baseos 36 M 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout:Install 3 Packages 2024-01-27T23:11:43.680 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:11:43.681 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 90 M 2024-01-27T23:11:43.681 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 96 M 2024-01-27T23:11:43.681 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:11:46.181 INFO:teuthology.orchestra.run.smithi194.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 3.4 MB/s | 10 MB 00:02 2024-01-27T23:11:46.876 INFO:teuthology.orchestra.run.smithi121.stdout:(1/3): kernel-4.18.0-536.el8.x86_64.rpm 3.4 MB/s | 10 MB 00:03 2024-01-27T23:11:49.148 INFO:teuthology.orchestra.run.smithi194.stdout:(2/3): kernel-core-4.18.0-536.el8.x86_64.rpm 7.3 MB/s | 44 MB 00:05 2024-01-27T23:11:49.481 INFO:teuthology.orchestra.run.smithi194.stdout:(3/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 5.8 MB/s | 36 MB 00:06 2024-01-27T23:11:49.482 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:11:49.482 INFO:teuthology.orchestra.run.smithi194.stdout:Total 14 MB/s | 90 MB 00:06 2024-01-27T23:11:49.850 INFO:teuthology.orchestra.run.smithi121.stdout:(2/3): kernel-core-4.18.0-536.el8.x86_64.rpm 7.3 MB/s | 44 MB 00:05 2024-01-27T23:11:50.163 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:11:50.573 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:11:50.573 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:11:50.775 INFO:teuthology.orchestra.run.smithi121.stdout:(3/3): kernel-modules-4.18.0-536.el8.x86_64.rpm 5.3 MB/s | 36 MB 00:06 2024-01-27T23:11:50.776 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:11:50.776 INFO:teuthology.orchestra.run.smithi121.stdout:Total 13 MB/s | 90 MB 00:07 2024-01-27T23:11:51.458 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:11:51.810 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:11:51.840 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:11:51.894 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:11:51.895 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:11:53.143 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:11:53.174 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:11:54.625 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:11:55.243 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-01-27T23:11:55.924 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:11:56.427 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-core-4.18.0-536.el8.x86_64 1/3 2024-01-27T23:11:58.803 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-01-27T23:11:59.185 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-01-27T23:11:59.983 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 1/3 2024-01-27T23:12:00.368 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-01-27T23:12:02.621 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-01-27T23:12:02.934 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:03.801 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 2/3 2024-01-27T23:12:04.091 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : kernel-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:24.817 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:24.840 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:25.213 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:25.213 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-01-27T23:12:25.213 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-01-27T23:12:25.759 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:25.759 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:12:25.759 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:12:25.759 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-01-27T23:12:25.759 INFO:teuthology.orchestra.run.smithi194.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-01-27T23:12:25.759 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:12:25.759 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:12:25.901 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-core-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:25.923 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:26.243 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T23:12:26.278 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: kernel-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:26.278 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-4.18.0-536.el8.x86_64 1/3 2024-01-27T23:12:26.278 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-core-4.18.0-536.el8.x86_64 2/3 2024-01-27T23:12:26.458 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-536.el8.x86_64 2024-01-27T23:12:26.509 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-01-27T23:12:26.509 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-01-27T23:12:26.510 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-27T23:12:26.510 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-27T23:12:26.510 INFO:teuthology.task.kernel:Installing distro kernel on host.b... 2024-01-27T23:12:26.510 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi194.front.sepia.ceph.com, path=None, version=distro) 2024-01-27T23:12:26.510 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T23:12:26.721 INFO:teuthology.orchestra.run.smithi194.stdout:kernel-4.18.0-536.el8.x86_64 2024-01-27T23:12:26.721 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-01-27T23:12:26.721 DEBUG:teuthology.orchestra.run.smithi194:> sudo rpm -qi grub2-tools 2024-01-27T23:12:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : kernel-modules-4.18.0-536.el8.x86_64 3/3 2024-01-27T23:12:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:12:26.754 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:12:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-4.18.0-536.el8.x86_64 kernel-core-4.18.0-536.el8.x86_64 2024-01-27T23:12:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: kernel-modules-4.18.0-536.el8.x86_64 2024-01-27T23:12:26.754 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:12:26.754 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Name : grub2-tools 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Epoch : 1 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Version : 2.02 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Release : 129.el8 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Architecture: x86_64 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Group : System Environment/Base 2024-01-27T23:12:26.755 INFO:teuthology.orchestra.run.smithi194.stdout:Size : 9265586 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:License : GPLv3+ 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Relocations : (not relocatable) 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Packager : CentOS BuildSystem 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Vendor : CentOS 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Summary : Support tools for GRUB. 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:Description : 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-27T23:12:26.756 INFO:teuthology.orchestra.run.smithi194.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-27T23:12:26.757 INFO:teuthology.orchestra.run.smithi194.stdout:hardware devices. 2024-01-27T23:12:26.757 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:12:26.757 INFO:teuthology.orchestra.run.smithi194.stdout:This subpackage provides tools for support of all platforms. 2024-01-27T23:12:26.757 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-27T23:12:26.757 INFO:teuthology.task.kernel:Updating grub on smithi194 to boot 4.18.0-536.el8.x86_64 2024-01-27T23:12:26.758 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-27T23:12:26.903 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T23:12:27.119 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-536.el8.x86_64 2024-01-27T23:12:27.119 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-01-27T23:12:27.119 INFO:teuthology.task.kernel:Not newest distro kernel. Current: 4.18.0-489.el8.x86_64 Expected: 4.18.0-536.el8.x86_64 2024-01-27T23:12:27.119 INFO:teuthology.task.kernel:Skipping firmware on distro kernel 2024-01-27T23:12:27.119 DEBUG:teuthology.task.kernel:src is distro, skipping download 2024-01-27T23:12:27.119 INFO:teuthology.task.kernel:Installing distro kernel on host.a... 2024-01-27T23:12:27.119 DEBUG:teuthology.task.kernel:install_kernel(remote=ubuntu@smithi121.front.sepia.ceph.com, path=None, version=distro) 2024-01-27T23:12:27.120 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q kernel | sort -rV | head -n 1 2024-01-27T23:12:27.330 INFO:teuthology.orchestra.run.smithi121.stdout:kernel-4.18.0-536.el8.x86_64 2024-01-27T23:12:27.330 DEBUG:teuthology.task.kernel:get_latest_image_version_rpm: 4.18.0-536.el8.x86_64 2024-01-27T23:12:27.330 DEBUG:teuthology.orchestra.run.smithi121:> sudo rpm -qi grub2-tools 2024-01-27T23:12:27.363 INFO:teuthology.orchestra.run.smithi121.stdout:Name : grub2-tools 2024-01-27T23:12:27.363 INFO:teuthology.orchestra.run.smithi121.stdout:Epoch : 1 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Version : 2.02 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Release : 129.el8 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Architecture: x86_64 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Install Date: Thu 04 May 2023 10:17:48 PM UTC 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Group : System Environment/Base 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Size : 9265586 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:License : GPLv3+ 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Signature : RSA/SHA256, Tue 19 Jul 2022 02:52:06 PM UTC, Key ID 05b555b38483c65d 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Source RPM : grub2-2.02-129.el8.src.rpm 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Build Date : Tue 19 Jul 2022 02:30:39 PM UTC 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Build Host : kbuilder.bsys.centos.org 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Relocations : (not relocatable) 2024-01-27T23:12:27.364 INFO:teuthology.orchestra.run.smithi121.stdout:Packager : CentOS BuildSystem 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:Vendor : CentOS 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:URL : http://www.gnu.org/software/grub/ 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:Summary : Support tools for GRUB. 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:Description : 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:The GRand Unified Bootloader (GRUB) is a highly configurable and 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:customizable bootloader with modular architecture. It supports a rich 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:variety of kernel formats, file systems, computer architectures and 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:hardware devices. 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:12:27.365 INFO:teuthology.orchestra.run.smithi121.stdout:This subpackage provides tools for support of all platforms. 2024-01-27T23:12:27.366 INFO:teuthology.task.kernel:Updating Grub Version: grub2 2024-01-27T23:12:27.366 INFO:teuthology.task.kernel:Updating grub on smithi121 to boot 4.18.0-536.el8.x86_64 2024-01-27T23:12:27.366 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-mkconfig -o /boot/grub2/grub.cfg 2024-01-27T23:12:28.153 INFO:teuthology.orchestra.run.smithi194.stderr:Generating grub configuration file ... 2024-01-27T23:12:29.056 INFO:teuthology.orchestra.run.smithi121.stderr:Generating grub configuration file ... 2024-01-27T23:12:30.138 INFO:teuthology.orchestra.run.smithi194.stderr:done 2024-01-27T23:12:30.140 DEBUG:teuthology.orchestra.run.smithi194:> mktemp 2024-01-27T23:12:30.163 INFO:teuthology.orchestra.run.smithi194.stdout:/tmp/tmp.a5WWvOxOLw 2024-01-27T23:12:30.163 DEBUG:teuthology.orchestra.run.smithi194:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.a5WWvOxOLw 2024-01-27T23:12:30.279 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 0666 /tmp/tmp.a5WWvOxOLw 2024-01-27T23:12:30.383 DEBUG:teuthology.orchestra.remote:smithi194:/tmp/tmp.a5WWvOxOLw is 6KB 2024-01-27T23:12:30.431 DEBUG:teuthology.orchestra.run.smithi194:> rm -fr /tmp/tmp.a5WWvOxOLw 2024-01-27T23:12:30.448 DEBUG:teuthology.orchestra.run.smithi194:> sudo /bin/ls /boot/loader/entries || true 2024-01-27T23:12:30.513 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-27T23:12:30.513 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-27T23:12:30.513 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-27T23:12:30.513 INFO:teuthology.orchestra.run.smithi194.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-01-27T23:12:30.514 DEBUG:teuthology.orchestra.run.smithi194:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-01-27T23:12:30.680 DEBUG:teuthology.orchestra.run.smithi194:> sudo shutdown -r now 2024-01-27T23:12:31.294 INFO:teuthology.orchestra.run.smithi121.stderr:done 2024-01-27T23:12:31.295 DEBUG:teuthology.orchestra.run.smithi121:> mktemp 2024-01-27T23:12:31.326 INFO:teuthology.orchestra.run.smithi121.stdout:/tmp/tmp.937MgIf2Sl 2024-01-27T23:12:31.326 DEBUG:teuthology.orchestra.run.smithi121:> sudo cp /boot/grub2/grub.cfg /tmp/tmp.937MgIf2Sl 2024-01-27T23:12:31.409 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 0666 /tmp/tmp.937MgIf2Sl 2024-01-27T23:12:31.551 DEBUG:teuthology.orchestra.remote:smithi121:/tmp/tmp.937MgIf2Sl is 6KB 2024-01-27T23:12:31.599 DEBUG:teuthology.orchestra.run.smithi121:> rm -fr /tmp/tmp.937MgIf2Sl 2024-01-27T23:12:31.617 DEBUG:teuthology.orchestra.run.smithi121:> sudo /bin/ls /boot/loader/entries || true 2024-01-27T23:12:31.694 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-0-rescue.conf 2024-01-27T23:12:31.694 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-348.el8.x86_64.conf 2024-01-27T23:12:31.694 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-489.el8.x86_64.conf 2024-01-27T23:12:31.694 INFO:teuthology.orchestra.run.smithi121.stdout:a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64.conf 2024-01-27T23:12:31.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo grub2-set-default a1b29d6c289747bdb22930598d39874a-4.18.0-536.el8.x86_64 2024-01-27T23:12:31.832 DEBUG:teuthology.orchestra.run.smithi121:> sudo shutdown -r now 2024-01-27T23:13:00.709 INFO:teuthology.task.kernel:Checking client host.b for new kernel version... 2024-01-27T23:13:00.709 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-27T23:13:00.710 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:13:01.836 INFO:teuthology.task.kernel:Checking client host.a for new kernel version... 2024-01-27T23:13:01.836 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-01-27T23:13:01.837 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:13:19.164 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-01-27T23:13:20.188 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.121 2024-01-27T23:13:28.172 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-27T23:13:28.173 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:13:29.190 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-01-27T23:13:29.190 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:13:31.228 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-01-27T23:13:43.240 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-27T23:13:43.240 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:14:01.664 DEBUG:teuthology.orchestra.remote:[Errno None] Unable to connect to port 22 on 172.21.15.194 2024-01-27T23:14:16.680 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-27T23:14:16.681 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:14:17.622 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-01-27T23:14:18.495 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-27T23:14:18.495 INFO:teuthology.task.kernel:Checking kernel version of host.b, want "4.18.0-536.el8.x86_64"... 2024-01-27T23:14:18.495 DEBUG:teuthology.orchestra.run.smithi194:> uname -r 2024-01-27T23:14:18.557 INFO:teuthology.orchestra.run.smithi194.stdout:4.18.0-536.el8.x86_64 2024-01-27T23:14:18.557 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-01-27T23:14:18.557 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-27T23:14:18.557 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-27T23:14:19.558 INFO:teuthology.task.kernel:Enabling kdb on host.b... 2024-01-27T23:14:19.558 DEBUG:teuthology.orchestra.run.smithi194:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-27T23:14:19.639 INFO:teuthology.orchestra.run.smithi194.stdout:ttyS1 2024-01-27T23:14:19.651 DEBUG:teuthology.parallel:result is None 2024-01-27T23:14:29.192 DEBUG:teuthology.orchestra.remote:timed out 2024-01-27T23:14:41.193 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-01-27T23:14:41.193 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:14:41.454 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-01-27T23:14:41.797 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-01-27T23:14:41.798 INFO:teuthology.task.kernel:Checking kernel version of host.a, want "4.18.0-536.el8.x86_64"... 2024-01-27T23:14:41.798 DEBUG:teuthology.orchestra.run.smithi121:> uname -r 2024-01-27T23:14:41.819 INFO:teuthology.orchestra.run.smithi121.stdout:4.18.0-536.el8.x86_64 2024-01-27T23:14:41.819 DEBUG:teuthology.task.kernel:current kernel version is 4.18.0-536.el8.x86_64 vs 4.18.0-536.el8.x86_64 2024-01-27T23:14:41.819 DEBUG:teuthology.task.kernel:utsrelease strings match, do not need to install 2024-01-27T23:14:41.819 DEBUG:teuthology.task.kernel:Distro of this test job: centos 2024-01-27T23:14:42.820 INFO:teuthology.task.kernel:Enabling kdb on host.a... 2024-01-27T23:14:42.820 DEBUG:teuthology.orchestra.run.smithi121:> echo ttyS1 | sudo tee /sys/module/kgdboc/parameters/kgdboc 2024-01-27T23:14:42.913 INFO:teuthology.orchestra.run.smithi121.stdout:ttyS1 2024-01-27T23:14:42.926 DEBUG:teuthology.parallel:result is None 2024-01-27T23:14:42.926 INFO:teuthology.run_tasks:Running task internal.base... 2024-01-27T23:14:42.933 INFO:teuthology.task.internal:Creating test directory... 2024-01-27T23:14:42.934 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-27T23:14:42.936 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest 2024-01-27T23:14:42.958 INFO:teuthology.run_tasks:Running task internal.archive_upload... 2024-01-27T23:14:42.990 INFO:teuthology.run_tasks:Running task internal.archive... 2024-01-27T23:14:43.022 INFO:teuthology.task.internal:Creating archive directory... 2024-01-27T23:14:43.022 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-27T23:14:43.025 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive 2024-01-27T23:14:43.076 INFO:teuthology.run_tasks:Running task internal.coredump... 2024-01-27T23:14:43.083 INFO:teuthology.task.internal:Enabling coredump saving... 2024-01-27T23:14:43.083 DEBUG:teuthology.orchestra.run.smithi121:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-27T23:14:43.086 DEBUG:teuthology.orchestra.run.smithi194:> install -d -m0755 -- /home/ubuntu/cephtest/archive/coredump && sudo sysctl -w kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core && echo kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core | sudo tee -a /etc/sysctl.conf 2024-01-27T23:14:43.123 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T23:14:43.128 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = /home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T23:14:43.133 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T23:14:43.139 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern=/home/ubuntu/cephtest/archive/coredump/%t.%p.core 2024-01-27T23:14:43.141 INFO:teuthology.run_tasks:Running task internal.sudo... 2024-01-27T23:14:43.147 INFO:teuthology.task.internal:Configuring sudo... 2024-01-27T23:14:43.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-27T23:14:43.177 DEBUG:teuthology.orchestra.run.smithi194:> sudo sed -i.orig.teuthology -e 's/^\([^#]*\) \(requiretty\)/\1 !\2/g' -e 's/^\([^#]*\) !\(visiblepw\)/\1 \2/g' /etc/sudoers 2024-01-27T23:14:43.211 INFO:teuthology.run_tasks:Running task internal.syslog... 2024-01-27T23:14:43.221 INFO:teuthology.task.internal.syslog:Starting syslog monitoring... 2024-01-27T23:14:43.221 DEBUG:teuthology.orchestra.run.smithi121:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-27T23:14:43.246 DEBUG:teuthology.orchestra.run.smithi194:> mkdir -p -m0755 -- /home/ubuntu/cephtest/archive/syslog 2024-01-27T23:14:43.270 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T23:14:43.349 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T23:14:43.397 DEBUG:teuthology.orchestra.run.smithi121:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T23:14:43.481 DEBUG:teuthology.orchestra.run.smithi121:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T23:14:43.523 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:14:43.524 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-27T23:14:43.589 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T23:14:43.634 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/kern.log 2024-01-27T23:14:43.695 DEBUG:teuthology.orchestra.run.smithi194:> install -m 666 /dev/null /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T23:14:43.782 DEBUG:teuthology.orchestra.run.smithi194:> sudo chcon system_u:object_r:var_log_t:s0 /home/ubuntu/cephtest/archive/syslog/misc.log 2024-01-27T23:14:43.827 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:14:43.827 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/rsyslog.d/80-cephtest.conf 2024-01-27T23:14:43.896 DEBUG:teuthology.orchestra.run.smithi121:> sudo service rsyslog restart 2024-01-27T23:14:43.898 DEBUG:teuthology.orchestra.run.smithi194:> sudo service rsyslog restart 2024-01-27T23:14:43.955 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T23:14:44.004 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T23:14:44.349 INFO:teuthology.run_tasks:Running task internal.timer... 2024-01-27T23:14:44.360 INFO:teuthology.task.internal:Starting timer... 2024-01-27T23:14:44.360 INFO:teuthology.run_tasks:Running task pcp... 2024-01-27T23:14:44.393 INFO:teuthology.run_tasks:Running task selinux... 2024-01-27T23:14:44.461 DEBUG:teuthology.task:Applying overrides for task selinux: {'whitelist': ['scontext=system_u:system_r:logrotate_t:s0']} 2024-01-27T23:14:44.461 DEBUG:teuthology.orchestra.run.smithi121:> sudo service auditd rotate 2024-01-27T23:14:44.544 INFO:teuthology.orchestra.run.smithi121.stdout:Rotating logs: 2024-01-27T23:14:44.546 DEBUG:teuthology.orchestra.run.smithi194:> sudo service auditd rotate 2024-01-27T23:14:44.620 INFO:teuthology.orchestra.run.smithi194.stdout:Rotating logs: 2024-01-27T23:14:44.621 DEBUG:teuthology.task.selinux:Getting current SELinux state 2024-01-27T23:14:44.622 DEBUG:teuthology.orchestra.run.smithi121:> /usr/sbin/getenforce 2024-01-27T23:14:44.678 INFO:teuthology.orchestra.run.smithi121.stdout:Permissive 2024-01-27T23:14:44.678 DEBUG:teuthology.orchestra.run.smithi194:> /usr/sbin/getenforce 2024-01-27T23:14:44.732 INFO:teuthology.orchestra.run.smithi194.stdout:Permissive 2024-01-27T23:14:44.732 DEBUG:teuthology.task.selinux:Existing SELinux modes: {'ubuntu@smithi121.front.sepia.ceph.com': 'permissive', 'ubuntu@smithi194.front.sepia.ceph.com': 'permissive'} 2024-01-27T23:14:44.733 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T23:14:44.759 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:14:44.760 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T23:14:44.786 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:14:44.786 INFO:teuthology.task.selinux:Putting SELinux into permissive mode 2024-01-27T23:14:44.786 DEBUG:teuthology.orchestra.run.smithi121:> sudo /usr/sbin/setenforce permissive 2024-01-27T23:14:44.826 DEBUG:teuthology.orchestra.run.smithi194:> sudo /usr/sbin/setenforce permissive 2024-01-27T23:14:44.857 INFO:teuthology.run_tasks:Running task ansible.cephlab... 2024-01-27T23:14:44.867 DEBUG:teuthology.repo_utils:Resetting repo at /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main to origin/main 2024-01-27T23:14:44.885 INFO:teuthology.task.ansible:Playbook: [{'import_playbook': 'ansible_managed.yml'}, {'import_playbook': 'teuthology.yml'}, {'hosts': 'testnodes', 'tasks': [{'set_fact': {'ran_from_cephlab_playbook': True}}]}, {'import_playbook': 'testnodes.yml'}, {'import_playbook': 'container-host.yml'}, {'import_playbook': 'cobbler.yml'}, {'import_playbook': 'paddles.yml'}, {'import_playbook': 'pulpito.yml'}, {'hosts': 'testnodes', 'become': True, 'tasks': [{'name': 'Touch /ceph-qa-ready', 'file': {'path': '/ceph-qa-ready', 'state': 'touch'}, 'when': 'ran_from_cephlab_playbook|bool'}]}] 2024-01-27T23:14:44.886 DEBUG:teuthology.task.ansible:Running ansible-playbook -v --extra-vars '{"ansible_ssh_user": "ubuntu"}' -i /etc/ansible/hosts --limit smithi121.front.sepia.ceph.com,smithi194.front.sepia.ceph.com /home/teuthworker/src/git.ceph.com_ceph-cm-ansible_main/cephlab.yml 2024-01-27T23:20:26.775 DEBUG:teuthology.task.ansible:Reconnecting to [Remote(name='ubuntu@smithi121.front.sepia.ceph.com'), Remote(name='ubuntu@smithi194.front.sepia.ceph.com')] 2024-01-27T23:20:26.777 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-01-27T23:20:26.778 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi121.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:20:26.856 DEBUG:teuthology.orchestra.run.smithi121:> true 2024-01-27T23:20:26.926 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi121.front.sepia.ceph.com' 2024-01-27T23:20:26.926 INFO:teuthology.orchestra.remote:Trying to reconnect to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-27T23:20:26.927 DEBUG:teuthology.orchestra.connection:{'hostname': 'smithi194.front.sepia.ceph.com', 'username': 'ubuntu', 'timeout': 60} 2024-01-27T23:20:27.003 DEBUG:teuthology.orchestra.run.smithi194:> true 2024-01-27T23:20:27.071 INFO:teuthology.orchestra.remote:Successfully reconnected to host 'ubuntu@smithi194.front.sepia.ceph.com' 2024-01-27T23:20:27.072 INFO:teuthology.run_tasks:Running task clock... 2024-01-27T23:20:27.082 INFO:teuthology.task.clock:Syncing clocks and checking initial clock skew... 2024-01-27T23:20:27.082 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-27T23:20:27.082 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T23:20:27.084 INFO:teuthology.orchestra.run:Running command with timeout 360 2024-01-27T23:20:27.084 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ntp.service || sudo systemctl stop ntpd.service || sudo systemctl stop chronyd.service ; sudo ntpd -gq || sudo chronyc makestep ; sudo systemctl start ntp.service || sudo systemctl start ntpd.service || sudo systemctl start chronyd.service ; PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T23:20:27.113 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-27T23:20:27.127 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-27T23:20:27.145 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntp.service: Unit ntp.service not loaded. 2024-01-27T23:20:27.159 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ntpd.service: Unit ntpd.service not loaded. 2024-01-27T23:20:27.186 INFO:teuthology.orchestra.run.smithi121.stderr:sudo: ntpd: command not found 2024-01-27T23:20:27.197 INFO:teuthology.orchestra.run.smithi121.stdout:506 Cannot talk to daemon 2024-01-27T23:20:27.211 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-27T23:20:27.224 INFO:teuthology.orchestra.run.smithi121.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-27T23:20:27.224 INFO:teuthology.orchestra.run.smithi194.stderr:sudo: ntpd: command not found 2024-01-27T23:20:27.236 INFO:teuthology.orchestra.run.smithi194.stdout:506 Cannot talk to daemon 2024-01-27T23:20:27.249 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntp.service: Unit ntp.service not found. 2024-01-27T23:20:27.262 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to start ntpd.service: Unit ntpd.service not found. 2024-01-27T23:20:27.297 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2024-01-27T23:20:27.301 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T23:20:27.301 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-01-27T23:20:27.301 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.301 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.301 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.301 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.321 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-01-27T23:20:27.324 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T23:20:27.325 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-01-27T23:20:27.325 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv01.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.325 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv02.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.325 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv03.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.325 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 6 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:20:27.326 INFO:teuthology.run_tasks:Running task pexec... 2024-01-27T23:20:27.336 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-27T23:20:27.337 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi121.front.sepia.ceph.com 2024-01-27T23:20:27.337 DEBUG:teuthology.orchestra.run.smithi121:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-27T23:20:27.337 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi194.front.sepia.ceph.com 2024-01-27T23:20:27.337 DEBUG:teuthology.orchestra.run.smithi194:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-27T23:20:28.117 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:22 ago on Sat 27 Jan 2024 11:19:05 PM UTC. 2024-01-27T23:20:28.146 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:22 ago on Sat 27 Jan 2024 11:19:05 PM UTC. 2024-01-27T23:20:30.083 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Resetting modules: 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:20:30.084 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:20:30.107 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout:Resetting modules: 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:20:30.108 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:20:30.338 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:20:30.359 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:20:31.187 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:26 ago on Sat 27 Jan 2024 11:19:05 PM UTC. 2024-01-27T23:20:31.225 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:26 ago on Sat 27 Jan 2024 11:19:05 PM UTC. 2024-01-27T23:20:33.168 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:20:33.203 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:20:33.203 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-01-27T23:20:33.203 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:20:33.203 INFO:teuthology.orchestra.run.smithi121.stdout:Installing group/module packages: 2024-01-27T23:20:33.203 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-27T23:20:33.203 INFO:teuthology.orchestra.run.smithi121.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout:Installing weak dependencies: 2024-01-27T23:20:33.204 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout:Installing module profiles: 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools/common 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout:Enabling module streams: 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout: container-tools rhel8 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-01-27T23:20:33.205 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:20:33.206 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 27 M 2024-01-27T23:20:33.206 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 89 M 2024-01-27T23:20:33.206 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:20:33.210 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout:Installing group/module packages: 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns x86_64 2:1.8.0-1.module_el8+712+4cd1bd69 appstream 1.0 M 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: buildah x86_64 1:1.32.0-1.module_el8+712+4cd1bd69 appstream 9.6 M 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman noarch 78-1.module_el8+712+4cd1bd69 appstream 750 k 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: crun x86_64 1.9.2-1.module_el8+712+4cd1bd69 appstream 241 k 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: netavark x86_64 2:1.8.0-2.module_el8+712+4cd1bd69 appstream 3.9 M 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman noarch 4.7.0-1.module_el8+712+4cd1bd69 appstream 178 k 2024-01-27T23:20:33.246 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo x86_64 2:1.13.3-1.module_el8+712+4cd1bd69 appstream 8.1 M 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox x86_64 0.0.99.4-6.module_el8+712+4cd1bd69 appstream 2.5 M 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: udica noarch 0.2.6-20.module_el8+712+4cd1bd69 appstream 53 k 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml noarch 0.1.14-5.git7dea353.el8 appstream 25 k 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg noarch 0.25-16.el8 appstream 94 k 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: yajl x86_64 2.1.0-12.el8 appstream 41 k 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout:Installing weak dependencies: 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs x86_64 3.15-3.module_el8.7.0+1216+b022c01d appstream 38 k 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout:Installing module profiles: 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools/common 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout:Enabling module streams: 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: container-tools rhel8 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:20:33.247 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:20:33.248 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:20:33.248 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-01-27T23:20:33.248 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:20:33.248 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 27 M 2024-01-27T23:20:33.248 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 89 M 2024-01-27T23:20:33.248 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:20:33.733 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.5 MB/s | 750 kB 00:00 2024-01-27T23:20:33.767 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.9 MB/s | 1.0 MB 00:00 2024-01-27T23:20:33.817 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 453 kB/s | 38 kB 00:00 2024-01-27T23:20:33.859 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.6 MB/s | 241 kB 00:00 2024-01-27T23:20:33.936 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): cockpit-podman-78-1.module_el8+712+4cd1 1.5 MB/s | 750 kB 00:00 2024-01-27T23:20:33.951 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 1.9 MB/s | 178 kB 00:00 2024-01-27T23:20:33.999 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): aardvark-dns-1.8.0-1.module_el8+712+4cd 1.8 MB/s | 1.0 MB 00:00 2024-01-27T23:20:34.026 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 330 kB/s | 25 kB 00:00 2024-01-27T23:20:34.027 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): criu-libs-3.15-3.module_el8.7.0+1216+b0 414 kB/s | 38 kB 00:00 2024-01-27T23:20:34.086 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): crun-1.9.2-1.module_el8+712+4cd1bd69.x8 2.7 MB/s | 241 kB 00:00 2024-01-27T23:20:34.159 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 11 MB/s | 9.6 MB 00:00 2024-01-27T23:20:34.170 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-podman-4.7.0-1.module_el8+712+4 2.1 MB/s | 178 kB 00:00 2024-01-27T23:20:34.185 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 592 kB/s | 94 kB 00:00 2024-01-27T23:20:34.244 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 9.2 MB/s | 3.9 MB 00:00 2024-01-27T23:20:34.253 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-pytoml-0.1.14-5.git7dea353.el8. 297 kB/s | 25 kB 00:00 2024-01-27T23:20:34.312 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): netavark-1.8.0-2.module_el8+712+4cd1bd6 14 MB/s | 3.9 MB 00:00 2024-01-27T23:20:34.319 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 702 kB/s | 53 kB 00:00 2024-01-27T23:20:34.337 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-pyxdg-0.25-16.el8.noarch.rpm 1.1 MB/s | 94 kB 00:00 2024-01-27T23:20:34.411 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 11 MB/s | 2.5 MB 00:00 2024-01-27T23:20:34.428 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 377 kB/s | 41 kB 00:00 2024-01-27T23:20:34.471 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): buildah-1.32.0-1.module_el8+712+4cd1bd6 9.3 MB/s | 9.6 MB 00:01 2024-01-27T23:20:34.555 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): udica-0.2.6-20.module_el8+712+4cd1bd69 631 kB/s | 53 kB 00:00 2024-01-27T23:20:34.604 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 18 MB/s | 8.1 MB 00:00 2024-01-27T23:20:34.604 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:20:34.604 INFO:teuthology.orchestra.run.smithi121.stdout:Total 19 MB/s | 27 MB 00:01 2024-01-27T23:20:34.655 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): toolbox-0.0.99.4-6.module_el8+712+4cd1 8.0 MB/s | 2.5 MB 00:00 2024-01-27T23:20:34.680 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): yajl-2.1.0-12.el8.x86_64.rpm 328 kB/s | 41 kB 00:00 2024-01-27T23:20:34.780 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): skopeo-1.13.3-1.module_el8+712+4cd1bd6 17 MB/s | 8.1 MB 00:00 2024-01-27T23:20:34.781 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:20:34.781 INFO:teuthology.orchestra.run.smithi194.stdout:Total 17 MB/s | 27 MB 00:01 2024-01-27T23:20:34.869 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:20:34.890 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:20:34.935 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:20:35.048 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:20:35.068 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:20:35.069 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:20:35.152 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:20:35.155 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:20:35.353 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:20:35.356 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:20:35.874 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:20:36.031 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:20:36.114 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-27T23:20:36.267 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-27T23:20:36.270 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : yajl-2.1.0-12.el8.x86_64 1/13 2024-01-27T23:20:36.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyxdg-0.25-16.el8.noarch 2/13 2024-01-27T23:20:36.504 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-27T23:20:36.653 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 3/13 2024-01-27T23:20:36.737 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-27T23:20:36.875 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-27T23:20:37.289 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-27T23:20:37.424 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 5/13 2024-01-27T23:20:38.282 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-27T23:20:38.406 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-27T23:20:38.529 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-27T23:20:38.679 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 7/13 2024-01-27T23:20:38.768 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-27T23:20:38.877 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 8/13 2024-01-27T23:20:39.071 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-27T23:20:39.186 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 9/13 2024-01-27T23:20:39.497 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-27T23:20:39.603 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 10/13 2024-01-27T23:20:40.420 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-27T23:20:40.486 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-27T23:20:40.647 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-27T23:20:40.703 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 12/13 2024-01-27T23:20:40.866 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-27T23:20:40.898 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-27T23:20:41.292 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-27T23:20:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-27T23:20:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-27T23:20:41.293 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-27T23:20:41.314 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 13/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86 1/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 3/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_ 4/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 5/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 6/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noa 7/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytoml-0.1.14-5.git7dea353.el8.noarch 8/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyxdg-0.25-16.el8.noarch 9/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 10/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 11/13 2024-01-27T23:20:41.315 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 12/13 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-27T23:20:41.826 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-27T23:20:41.827 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-27T23:20:41.827 INFO:teuthology.orchestra.run.smithi194.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.827 INFO:teuthology.orchestra.run.smithi194.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.827 INFO:teuthology.orchestra.run.smithi194.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-27T23:20:41.827 INFO:teuthology.orchestra.run.smithi194.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-27T23:20:41.827 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:20:41.827 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : yajl-2.1.0-12.el8.x86_64 13/13 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: aardvark-dns-2:1.8.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: buildah-1:1.32.0-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: cockpit-podman-78-1.module_el8+712+4cd1bd69.noarch 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: criu-libs-3.15-3.module_el8.7.0+1216+b022c01d.x86_64 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: crun-1.9.2-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: netavark-2:1.8.0-2.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: python3-podman-4.7.0-1.module_el8+712+4cd1bd69.noarch 2024-01-27T23:20:41.872 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytoml-0.1.14-5.git7dea353.el8.noarch 2024-01-27T23:20:41.873 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyxdg-0.25-16.el8.noarch 2024-01-27T23:20:41.873 INFO:teuthology.orchestra.run.smithi121.stdout: skopeo-2:1.13.3-1.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.873 INFO:teuthology.orchestra.run.smithi121.stdout: toolbox-0.0.99.4-6.module_el8+712+4cd1bd69.x86_64 2024-01-27T23:20:41.873 INFO:teuthology.orchestra.run.smithi121.stdout: udica-0.2.6-20.module_el8+712+4cd1bd69.noarch 2024-01-27T23:20:41.873 INFO:teuthology.orchestra.run.smithi121.stdout: yajl-2.1.0-12.el8.x86_64 2024-01-27T23:20:41.873 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:20:41.873 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:20:42.126 DEBUG:teuthology.parallel:result is None 2024-01-27T23:20:42.147 DEBUG:teuthology.parallel:result is None 2024-01-27T23:20:42.148 INFO:teuthology.run_tasks:Running task pexec... 2024-01-27T23:20:42.155 INFO:teuthology.task.pexec:Executing custom commands... 2024-01-27T23:20:42.155 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi121.front.sepia.ceph.com 2024-01-27T23:20:42.156 DEBUG:teuthology.orchestra.run.smithi121:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-27T23:20:42.156 INFO:teuthology.task.pexec:Running commands on host ubuntu@smithi194.front.sepia.ceph.com 2024-01-27T23:20:42.156 DEBUG:teuthology.orchestra.run.smithi194:> TESTDIR=/home/ubuntu/cephtest bash -s 2024-01-27T23:20:42.333 DEBUG:teuthology.parallel:result is None 2024-01-27T23:20:42.397 DEBUG:teuthology.parallel:result is None 2024-01-27T23:20:42.397 INFO:teuthology.run_tasks:Running task install... 2024-01-27T23:20:42.406 DEBUG:teuthology.task.install:project ceph 2024-01-27T23:20:42.406 DEBUG:teuthology.task.install:INSTALL overrides: {'ceph': {'flavor': 'default', 'sha1': 'a72a5dace2353a54147d37bd2699a2a482507966'}} 2024-01-27T23:20:42.406 DEBUG:teuthology.task.install:config {'exclude_packages': ['ceph-volume'], 'tag': 'v16.2.4', 'flavor': 'default', 'sha1': 'a72a5dace2353a54147d37bd2699a2a482507966'} 2024-01-27T23:20:42.406 INFO:teuthology.task.install:Using flavor: default 2024-01-27T23:20:42.413 DEBUG:teuthology.task.install:Package list is: {'deb': ['ceph', 'cephadm', 'ceph-mds', 'ceph-mgr', 'ceph-common', 'ceph-fuse', 'ceph-test', 'radosgw', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'libcephfs2', 'libcephfs-dev', 'librados2', 'librbd1', 'rbd-fuse'], 'rpm': ['ceph-radosgw', 'ceph-test', 'ceph', 'ceph-base', 'cephadm', 'ceph-immutable-object-cache', 'ceph-mgr', 'ceph-mgr-dashboard', 'ceph-mgr-diskprediction-local', 'ceph-mgr-rook', 'ceph-mgr-cephadm', 'ceph-fuse', 'librados-devel', 'libcephfs2', 'libcephfs-devel', 'librados2', 'librbd1', 'python3-rados', 'python3-rgw', 'python3-cephfs', 'python3-rbd', 'rbd-fuse', 'rbd-mirror', 'rbd-nbd']} 2024-01-27T23:20:42.413 INFO:teuthology.task.install:extra packages: [] 2024-01-27T23:20:42.414 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a72a5dace2353a54147d37bd2699a2a482507966', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-27T23:20:42.414 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-27T23:20:42.414 INFO:teuthology.packaging:ref: None 2024-01-27T23:20:42.414 INFO:teuthology.packaging:tag: v16.2.4 2024-01-27T23:20:42.414 INFO:teuthology.packaging:branch: None 2024-01-27T23:20:42.414 INFO:teuthology.packaging:sha1: a72a5dace2353a54147d37bd2699a2a482507966 2024-01-27T23:20:42.693 DEBUG:teuthology.repo_utils:git ls-remote https://github.com/ceph/ceph-ci v16.2.4^{} -> 3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-27T23:20:42.693 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-27T23:20:42.696 DEBUG:teuthology.task.install.rpm:_update_package_list_and_install: config is {'branch': None, 'cleanup': None, 'debuginfo': None, 'downgrade_packages': [], 'exclude_packages': ['ceph-volume'], 'extra_packages': [], 'extra_system_packages': [], 'extras': None, 'enable_coprs': [], 'flavor': 'default', 'install_ceph_packages': True, 'packages': {}, 'project': 'ceph', 'repos_only': False, 'sha1': 'a72a5dace2353a54147d37bd2699a2a482507966', 'tag': 'v16.2.4', 'wait_for_package': False} 2024-01-27T23:20:42.696 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-27T23:20:42.696 INFO:teuthology.packaging:ref: None 2024-01-27T23:20:42.696 INFO:teuthology.packaging:tag: v16.2.4 2024-01-27T23:20:42.696 INFO:teuthology.packaging:branch: None 2024-01-27T23:20:42.696 INFO:teuthology.packaging:sha1: a72a5dace2353a54147d37bd2699a2a482507966 2024-01-27T23:20:42.696 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-27T23:20:42.837 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-27T23:20:42.837 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-27T23:20:42.849 INFO:teuthology.task.install.rpm:Pulling from https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/ 2024-01-27T23:20:42.849 INFO:teuthology.task.install.rpm:Package version is 16.2.4-0 2024-01-27T23:20:42.880 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-27T23:20:42.880 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:20:42.880 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-27T23:20:42.889 INFO:teuthology.packaging:Writing yum repo: [ceph] name=ceph packages for $basearch baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/$basearch enabled=1 gpgcheck=0 type=rpm-md [ceph-noarch] name=ceph noarch packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/noarch enabled=1 gpgcheck=0 type=rpm-md [ceph-source] name=ceph source packages baseurl=https://chacra.ceph.com/r/ceph/pacific/3cbe25cde3cfa028984618ad32de9edc4c1eaed0/centos/8/flavors/default/SRPMS enabled=1 gpgcheck=0 type=rpm-md 2024-01-27T23:20:42.889 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:20:42.889 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/etc/yum.repos.d/ceph.repo 2024-01-27T23:20:42.907 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-27T23:20:42.908 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-27T23:20:42.908 INFO:teuthology.packaging:ref: None 2024-01-27T23:20:42.908 INFO:teuthology.packaging:tag: v16.2.4 2024-01-27T23:20:42.908 INFO:teuthology.packaging:branch: None 2024-01-27T23:20:42.908 INFO:teuthology.packaging:sha1: a72a5dace2353a54147d37bd2699a2a482507966 2024-01-27T23:20:42.908 DEBUG:teuthology.orchestra.run.smithi121:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-27T23:20:42.917 INFO:teuthology.task.install.rpm:Installing packages: ceph-radosgw, ceph-test, ceph, ceph-base, cephadm, ceph-immutable-object-cache, ceph-mgr, ceph-mgr-dashboard, ceph-mgr-diskprediction-local, ceph-mgr-rook, ceph-mgr-cephadm, ceph-fuse, librados-devel, libcephfs2, libcephfs-devel, librados2, librbd1, python3-rados, python3-rgw, python3-cephfs, python3-rbd, rbd-fuse, rbd-mirror, rbd-nbd on remote rpm x86_64 2024-01-27T23:20:42.917 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-27T23:20:42.917 INFO:teuthology.packaging:ref: None 2024-01-27T23:20:42.917 INFO:teuthology.packaging:tag: v16.2.4 2024-01-27T23:20:42.917 INFO:teuthology.packaging:branch: None 2024-01-27T23:20:42.917 INFO:teuthology.packaging:sha1: a72a5dace2353a54147d37bd2699a2a482507966 2024-01-27T23:20:42.918 DEBUG:teuthology.orchestra.run.smithi194:> if test -f /etc/yum.repos.d/ceph.repo ; then sudo sed -i -e ':a;N;$!ba;s/enabled=1\ngpg/enabled=1\npriority=1\ngpg/g' -e 's;ref/[a-zA-Z0-9_-]*/;ref/v16.2.4/;g' /etc/yum.repos.d/ceph.repo ; fi 2024-01-27T23:20:42.975 DEBUG:teuthology.orchestra.run.smithi121:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-27T23:20:42.985 DEBUG:teuthology.orchestra.run.smithi194:> sudo touch -a /etc/yum/pluginconf.d/priorities.conf ; test -e /etc/yum/pluginconf.d/priorities.conf.orig || sudo cp -af /etc/yum/pluginconf.d/priorities.conf /etc/yum/pluginconf.d/priorities.conf.orig 2024-01-27T23:20:43.052 DEBUG:teuthology.orchestra.run.smithi121:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-27T23:20:43.062 DEBUG:teuthology.orchestra.run.smithi194:> grep check_obsoletes /etc/yum/pluginconf.d/priorities.conf && sudo sed -i 's/check_obsoletes.*0/check_obsoletes = 1/g' /etc/yum/pluginconf.d/priorities.conf || echo 'check_obsoletes = 1' | sudo tee -a /etc/yum/pluginconf.d/priorities.conf 2024-01-27T23:20:43.118 INFO:teuthology.orchestra.run.smithi121.stdout:check_obsoletes = 1 2024-01-27T23:20:43.119 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum clean all 2024-01-27T23:20:43.128 INFO:teuthology.orchestra.run.smithi194.stdout:check_obsoletes = 1 2024-01-27T23:20:43.130 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum clean all 2024-01-27T23:20:43.384 INFO:teuthology.orchestra.run.smithi121.stdout:76 files removed 2024-01-27T23:20:43.399 INFO:teuthology.orchestra.run.smithi194.stdout:76 files removed 2024-01-27T23:20:43.411 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-radosgw 2024-01-27T23:20:43.430 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-radosgw 2024-01-27T23:20:43.903 INFO:teuthology.orchestra.run.smithi121.stdout:ceph packages for x86_64 900 kB/s | 74 kB 00:00 2024-01-27T23:20:43.914 INFO:teuthology.orchestra.run.smithi194.stdout:ceph packages for x86_64 943 kB/s | 74 kB 00:00 2024-01-27T23:20:44.045 INFO:teuthology.orchestra.run.smithi121.stdout:ceph noarch packages 147 kB/s | 15 kB 00:00 2024-01-27T23:20:44.064 INFO:teuthology.orchestra.run.smithi194.stdout:ceph noarch packages 135 kB/s | 15 kB 00:00 2024-01-27T23:20:44.155 INFO:teuthology.orchestra.run.smithi121.stdout:ceph source packages 22 kB/s | 1.7 kB 00:00 2024-01-27T23:20:44.157 INFO:teuthology.orchestra.run.smithi194.stdout:ceph source packages 28 kB/s | 1.7 kB 00:00 2024-01-27T23:20:45.582 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - AppStream 19 MB/s | 27 MB 00:01 2024-01-27T23:20:45.764 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - AppStream 17 MB/s | 27 MB 00:01 2024-01-27T23:20:55.080 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - BaseOS 2.2 MB/s | 10 MB 00:04 2024-01-27T23:20:55.091 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - BaseOS 2.1 MB/s | 10 MB 00:04 2024-01-27T23:20:56.947 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras 69 kB/s | 18 kB 00:00 2024-01-27T23:20:57.100 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras 42 kB/s | 18 kB 00:00 2024-01-27T23:20:57.417 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - Extras common packages 33 kB/s | 6.9 kB 00:00 2024-01-27T23:20:57.448 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - Extras common packages 18 kB/s | 6.9 kB 00:00 2024-01-27T23:20:58.057 INFO:teuthology.orchestra.run.smithi121.stdout:CentOS Stream 8 - PowerTools 9.2 MB/s | 4.8 MB 00:00 2024-01-27T23:20:58.677 INFO:teuthology.orchestra.run.smithi194.stdout:CentOS Stream 8 - PowerTools 4.1 MB/s | 4.8 MB 00:01 2024-01-27T23:20:59.092 INFO:teuthology.orchestra.run.smithi121.stdout:Copr repo for python3-asyncssh owned by ceph 24 kB/s | 3.5 kB 00:00 2024-01-27T23:20:59.736 INFO:teuthology.orchestra.run.smithi194.stdout:Copr repo for python3-asyncssh owned by ceph 21 kB/s | 3.5 kB 00:00 2024-01-27T23:21:00.151 INFO:teuthology.orchestra.run.smithi121.stdout:Extra Packages for Enterprise Linux 17 MB/s | 16 MB 00:00 2024-01-27T23:21:01.173 INFO:teuthology.orchestra.run.smithi194.stdout:Extra Packages for Enterprise Linux 12 MB/s | 16 MB 00:01 2024-01-27T23:21:03.943 INFO:teuthology.orchestra.run.smithi121.stdout:lab-extras 335 kB/s | 24 kB 00:00 2024-01-27T23:21:04.937 INFO:teuthology.orchestra.run.smithi194.stdout:lab-extras 395 kB/s | 24 kB 00:00 2024-01-27T23:21:07.413 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrading: 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:21:07.416 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-27T23:21:07.417 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:21:07.418 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:21:07.419 INFO:teuthology.orchestra.run.smithi121.stdout:Install 25 Packages 2024-01-27T23:21:07.419 INFO:teuthology.orchestra.run.smithi121.stdout:Upgrade 2 Packages 2024-01-27T23:21:07.419 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:07.419 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 59 M 2024-01-27T23:21:07.419 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:21:08.218 INFO:teuthology.orchestra.run.smithi121.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 36 MB/s | 6.0 MB 00:00 2024-01-27T23:21:08.269 INFO:teuthology.orchestra.run.smithi121.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 497 kB/s | 25 kB 00:00 2024-01-27T23:21:08.302 INFO:teuthology.orchestra.run.smithi121.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 23 MB/s | 801 kB 00:00 2024-01-27T23:21:08.336 INFO:teuthology.orchestra.run.smithi121.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-27T23:21:08.511 INFO:teuthology.orchestra.run.smithi121.stdout:(5/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 28 MB/s | 13 MB 00:00 2024-01-27T23:21:08.515 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:21:08.518 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:21:08.518 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-27T23:21:08.518 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:21:08.518 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:21:08.518 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw x86_64 2:16.2.4-0.el8 ceph 13 M 2024-01-27T23:21:08.518 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrading: 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: librados2 x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: librbd1 x86_64 2:16.2.4-0.el8 ceph 4.0 M 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base x86_64 2:16.2.4-0.el8 ceph 6.0 M 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common x86_64 2:16.2.4-0.el8 ceph 24 M 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux x86_64 2:16.2.4-0.el8 ceph 25 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: daxctl-libs x86_64 71.1-7.el8 baseos 42 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: fmt x86_64 6.2.1-1.el8 epel 116 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs x86_64 1:2.7-9.el8 epel 306 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb x86_64 1.22-1.el8 epel 181 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2 x86_64 2:16.2.4-0.el8 ceph 801 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: liboath x86_64 2.6.2-3.el8 epel 59 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj x86_64 1.6.1-1.el8 appstream 145 k 2024-01-27T23:21:08.519 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq x86_64 0.9.0-5.el8 baseos 47 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1 x86_64 2:16.2.4-0.el8 ceph 486 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka x86_64 1.6.1-1.el8 appstream 677 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2 x86_64 2:16.2.4-0.el8 ceph 4.1 M 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind x86_64 1.3.1-3.el8 epel 75 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust x86_64 2.8.1-11.el8 appstream 259 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: ndctl-libs x86_64 71.1-7.el8 baseos 80 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse x86_64 2:16.2.4-0.el8 ceph 44 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common x86_64 2:16.2.4-0.el8 ceph 80 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs x86_64 2:16.2.4-0.el8 ceph 213 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable noarch 0.7.2-14.el8 appstream 44 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados x86_64 2:16.2.4-0.el8 ceph 387 k 2024-01-27T23:21:08.520 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd x86_64 2:16.2.4-0.el8 ceph 368 k 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw x86_64 2:16.2.4-0.el8 ceph 114 k 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout:Install 25 Packages 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout:Upgrade 2 Packages 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:08.521 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 59 M 2024-01-27T23:21:08.522 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:21:08.570 INFO:teuthology.orchestra.run.smithi121.stdout:(6/27): librgw2-16.2.4-0.el8.x86_64.rpm 17 MB/s | 4.1 MB 00:00 2024-01-27T23:21:08.627 INFO:teuthology.orchestra.run.smithi121.stdout:(7/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 387 kB/s | 44 kB 00:00 2024-01-27T23:21:08.645 INFO:teuthology.orchestra.run.smithi121.stdout:(8/27): python3-ceph-common-16.2.4-0.el8.x86_64 1.0 MB/s | 80 kB 00:00 2024-01-27T23:21:08.679 INFO:teuthology.orchestra.run.smithi121.stdout:(9/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 4.0 MB/s | 213 kB 00:00 2024-01-27T23:21:08.704 INFO:teuthology.orchestra.run.smithi121.stdout:(10/27): python3-rados-16.2.4-0.el8.x86_64.rpm 6.5 MB/s | 387 kB 00:00 2024-01-27T23:21:08.730 INFO:teuthology.orchestra.run.smithi121.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-27T23:21:08.755 INFO:teuthology.orchestra.run.smithi121.stdout:(12/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 4.8 MB/s | 368 kB 00:00 2024-01-27T23:21:09.047 INFO:teuthology.orchestra.run.smithi121.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 24 MB/s | 24 MB 00:00 2024-01-27T23:21:09.188 INFO:teuthology.orchestra.run.smithi194.stdout:(1/27): ceph-base-16.2.4-0.el8.x86_64.rpm 50 MB/s | 6.0 MB 00:00 2024-01-27T23:21:09.205 INFO:teuthology.orchestra.run.smithi194.stdout:(2/27): ceph-selinux-16.2.4-0.el8.x86_64.rpm 1.5 MB/s | 25 kB 00:00 2024-01-27T23:21:09.239 INFO:teuthology.orchestra.run.smithi194.stdout:(3/27): libcephfs2-16.2.4-0.el8.x86_64.rpm 24 MB/s | 801 kB 00:00 2024-01-27T23:21:09.272 INFO:teuthology.orchestra.run.smithi194.stdout:(4/27): libradosstriper1-16.2.4-0.el8.x86_64.rp 14 MB/s | 486 kB 00:00 2024-01-27T23:21:09.339 INFO:teuthology.orchestra.run.smithi121.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 239 kB/s | 145 kB 00:00 2024-01-27T23:21:09.364 INFO:teuthology.orchestra.run.smithi194.stdout:(5/27): librgw2-16.2.4-0.el8.x86_64.rpm 44 MB/s | 4.1 MB 00:00 2024-01-27T23:21:09.415 INFO:teuthology.orchestra.run.smithi194.stdout:(6/27): python3-ceph-argparse-16.2.4-0.el8.x86_ 893 kB/s | 44 kB 00:00 2024-01-27T23:21:09.439 INFO:teuthology.orchestra.run.smithi121.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 440 kB/s | 44 kB 00:00 2024-01-27T23:21:09.440 INFO:teuthology.orchestra.run.smithi194.stdout:(7/27): python3-ceph-common-16.2.4-0.el8.x86_64 3.1 MB/s | 80 kB 00:00 2024-01-27T23:21:09.465 INFO:teuthology.orchestra.run.smithi194.stdout:(8/27): python3-cephfs-16.2.4-0.el8.x86_64.rpm 8.4 MB/s | 213 kB 00:00 2024-01-27T23:21:09.481 INFO:teuthology.orchestra.run.smithi121.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 599 kB/s | 259 kB 00:00 2024-01-27T23:21:09.499 INFO:teuthology.orchestra.run.smithi194.stdout:(9/27): python3-rados-16.2.4-0.el8.x86_64.rpm 11 MB/s | 387 kB 00:00 2024-01-27T23:21:09.506 INFO:teuthology.orchestra.run.smithi121.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 901 kB/s | 677 kB 00:00 2024-01-27T23:21:09.524 INFO:teuthology.orchestra.run.smithi194.stdout:(10/27): python3-rbd-16.2.4-0.el8.x86_64.rpm 14 MB/s | 368 kB 00:00 2024-01-27T23:21:09.549 INFO:teuthology.orchestra.run.smithi194.stdout:(11/27): python3-rgw-16.2.4-0.el8.x86_64.rpm 4.5 MB/s | 114 kB 00:00 2024-01-27T23:21:09.692 INFO:teuthology.orchestra.run.smithi194.stdout:(12/27): ceph-radosgw-16.2.4-0.el8.x86_64.rpm 20 MB/s | 13 MB 00:00 2024-01-27T23:21:09.790 INFO:teuthology.orchestra.run.smithi121.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 120 kB/s | 42 kB 00:00 2024-01-27T23:21:09.824 INFO:teuthology.orchestra.run.smithi121.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 137 kB/s | 47 kB 00:00 2024-01-27T23:21:09.849 INFO:teuthology.orchestra.run.smithi121.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 234 kB/s | 80 kB 00:00 2024-01-27T23:21:09.975 INFO:teuthology.orchestra.run.smithi194.stdout:(13/27): ceph-common-16.2.4-0.el8.x86_64.rpm 26 MB/s | 24 MB 00:00 2024-01-27T23:21:10.000 INFO:teuthology.orchestra.run.smithi194.stdout:(14/27): libpmemobj-1.6.1-1.el8.x86_64.rpm 322 kB/s | 145 kB 00:00 2024-01-27T23:21:10.101 INFO:teuthology.orchestra.run.smithi194.stdout:(15/27): python3-prettytable-0.7.2-14.el8.noarc 439 kB/s | 44 kB 00:00 2024-01-27T23:21:10.107 INFO:teuthology.orchestra.run.smithi121.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 366 kB/s | 116 kB 00:00 2024-01-27T23:21:10.175 INFO:teuthology.orchestra.run.smithi121.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 884 kB/s | 59 kB 00:00 2024-01-27T23:21:10.200 INFO:teuthology.orchestra.run.smithi121.stdout:(23/27): leveldb-1.22-1.el8.x86_64.rpm 516 kB/s | 181 kB 00:00 2024-01-27T23:21:10.225 INFO:teuthology.orchestra.run.smithi121.stdout:(24/27): gperftools-libs-2.7-9.el8.x86_64.rpm 763 kB/s | 306 kB 00:00 2024-01-27T23:21:10.250 INFO:teuthology.orchestra.run.smithi121.stdout:(25/27): libunwind-1.3.1-3.el8.x86_64.rpm 995 kB/s | 75 kB 00:00 2024-01-27T23:21:10.334 INFO:teuthology.orchestra.run.smithi121.stdout:(26/27): librados2-16.2.4-0.el8.x86_64.rpm 28 MB/s | 3.7 MB 00:00 2024-01-27T23:21:10.359 INFO:teuthology.orchestra.run.smithi194.stdout:(16/27): lttng-ust-2.8.1-11.el8.x86_64.rpm 677 kB/s | 259 kB 00:00 2024-01-27T23:21:10.384 INFO:teuthology.orchestra.run.smithi121.stdout:(27/27): librbd1-16.2.4-0.el8.x86_64.rpm 26 MB/s | 4.0 MB 00:00 2024-01-27T23:21:10.385 INFO:teuthology.orchestra.run.smithi194.stdout:(17/27): librdkafka-1.6.1-1.el8.x86_64.rpm 977 kB/s | 677 kB 00:00 2024-01-27T23:21:10.389 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:21:10.389 INFO:teuthology.orchestra.run.smithi121.stdout:Total 20 MB/s | 59 MB 00:02 2024-01-27T23:21:10.435 INFO:teuthology.orchestra.run.smithi194.stdout:(18/27): daxctl-libs-71.1-7.el8.x86_64.rpm 126 kB/s | 42 kB 00:00 2024-01-27T23:21:10.456 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:21:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:21:10.504 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:21:10.718 INFO:teuthology.orchestra.run.smithi194.stdout:(19/27): librabbitmq-0.9.0-5.el8.x86_64.rpm 131 kB/s | 47 kB 00:00 2024-01-27T23:21:10.744 INFO:teuthology.orchestra.run.smithi194.stdout:(20/27): ndctl-libs-71.1-7.el8.x86_64.rpm 223 kB/s | 80 kB 00:00 2024-01-27T23:21:10.819 INFO:teuthology.orchestra.run.smithi194.stdout:(21/27): fmt-6.2.1-1.el8.x86_64.rpm 303 kB/s | 116 kB 00:00 2024-01-27T23:21:10.844 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:21:10.847 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:21:10.944 INFO:teuthology.orchestra.run.smithi194.stdout:(22/27): liboath-2.6.2-3.el8.x86_64.rpm 470 kB/s | 59 kB 00:00 2024-01-27T23:21:11.094 INFO:teuthology.orchestra.run.smithi194.stdout:(23/27): libunwind-1.3.1-3.el8.x86_64.rpm 500 kB/s | 75 kB 00:00 2024-01-27T23:21:11.136 INFO:teuthology.orchestra.run.smithi194.stdout:(24/27): leveldb-1.22-1.el8.x86_64.rpm 461 kB/s | 181 kB 00:00 2024-01-27T23:21:11.203 INFO:teuthology.orchestra.run.smithi194.stdout:(25/27): librados2-16.2.4-0.el8.x86_64.rpm 34 MB/s | 3.7 MB 00:00 2024-01-27T23:21:11.270 INFO:teuthology.orchestra.run.smithi194.stdout:(26/27): librbd1-16.2.4-0.el8.x86_64.rpm 30 MB/s | 4.0 MB 00:00 2024-01-27T23:21:11.312 INFO:teuthology.orchestra.run.smithi194.stdout:(27/27): gperftools-libs-2.7-9.el8.x86_64.rpm 516 kB/s | 306 kB 00:00 2024-01-27T23:21:11.315 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:21:11.315 INFO:teuthology.orchestra.run.smithi194.stdout:Total 21 MB/s | 59 MB 00:02 2024-01-27T23:21:11.383 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:21:11.431 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:21:11.431 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:21:11.784 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:21:11.787 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:21:12.092 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:21:12.898 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-27T23:21:13.095 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-27T23:21:13.285 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-27T23:21:13.585 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:21:13.696 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-27T23:21:13.808 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fmt-6.2.1-1.el8.x86_64 1/29 2024-01-27T23:21:13.918 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-27T23:21:14.010 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-27T23:21:14.053 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : liboath-2.6.2-3.el8.x86_64 2/29 2024-01-27T23:21:14.218 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-27T23:21:14.250 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-27T23:21:14.426 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-27T23:21:14.480 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-27T23:21:14.630 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: lttng-ust-2.8.1-11.el8.x86_64 3/29 2024-01-27T23:21:14.718 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-27T23:21:14.851 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-27T23:21:14.946 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-2:16.2.4-0.el8.x86_64 4/29 2024-01-27T23:21:15.184 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rados-2:16.2.4-0.el8.x86_64 5/29 2024-01-27T23:21:15.262 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-27T23:21:15.343 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-27T23:21:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs2-2:16.2.4-0.el8.x86_64 6/29 2024-01-27T23:21:15.451 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-27T23:21:15.533 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-27T23:21:15.602 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librabbitmq-0.9.0-5.el8.x86_64 7/29 2024-01-27T23:21:15.731 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-27T23:21:15.911 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-27T23:21:15.977 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-27T23:21:16.177 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librdkafka-1.6.1-1.el8.x86_64 8/29 2024-01-27T23:21:16.181 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-27T23:21:16.343 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-27T23:21:16.351 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-27T23:21:16.415 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librgw2-2:16.2.4-0.el8.x86_64 9/29 2024-01-27T23:21:16.543 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-27T23:21:16.620 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-27T23:21:16.622 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : leveldb-1.22-1.el8.x86_64 10/29 2024-01-27T23:21:16.792 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-27T23:21:16.814 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-27T23:21:16.880 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: daxctl-libs-71.1-7.el8.x86_64 11/29 2024-01-27T23:21:17.073 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 12/29 2024-01-27T23:21:17.206 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-27T23:21:17.251 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cephfs-2:16.2.4-0.el8.x86_64 13/29 2024-01-27T23:21:17.409 INFO:teuthology.orchestra.run.smithi121.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-27T23:21:17.451 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-27T23:21:17.518 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-27T23:21:17.523 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ndctl-libs-71.1-7.el8.x86_64 14/29 2024-01-27T23:21:17.659 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-27T23:21:17.704 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-27T23:21:17.935 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-27T23:21:18.083 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-27T23:21:18.092 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libpmemobj-1.6.1-1.el8.x86_64 15/29 2024-01-27T23:21:18.254 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-27T23:21:18.301 INFO:teuthology.orchestra.run.smithi194.stdout: Upgrading : librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-27T23:21:18.396 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-2:16.2.4-0.el8.x86_64 16/29 2024-01-27T23:21:18.443 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-27T23:21:18.658 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-27T23:21:18.712 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rbd-2:16.2.4-0.el8.x86_64 17/29 2024-01-27T23:21:18.826 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-27T23:21:18.911 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rgw-2:16.2.4-0.el8.x86_64 18/29 2024-01-27T23:21:19.042 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-27T23:21:19.100 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-27T23:21:19.158 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libradosstriper1-2:16.2.4-0.el8.x86_64 19/29 2024-01-27T23:21:19.378 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libunwind-1.3.1-3.el8.x86_64 20/29 2024-01-27T23:21:19.576 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : gperftools-libs-1:2.7-9.el8.x86_64 21/29 2024-01-27T23:21:19.760 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-prettytable-0.7.2-14.el8.noarch 22/29 2024-01-27T23:21:19.993 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-ceph-common-2:16.2.4-0.el8.x86_64 23/29 2024-01-27T23:21:21.617 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-27T23:21:21.875 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-27T23:21:22.490 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-27T23:21:22.567 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-27T23:21:22.684 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-27T23:21:22.867 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-27T23:21:23.003 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-27T23:21:23.132 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-27T23:21:23.418 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-common-2:16.2.4-0.el8.x86_64 24/29 2024-01-27T23:21:24.096 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-27T23:21:24.233 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-base-2:16.2.4-0.el8.x86_64 25/29 2024-01-27T23:21:24.400 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-27T23:21:42.271 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-27T23:21:42.271 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /sys 2024-01-27T23:21:42.271 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /proc 2024-01-27T23:21:42.271 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2024-01-27T23:21:42.272 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /run 2024-01-27T23:21:42.272 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /mnt 2024-01-27T23:21:42.272 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /var/tmp 2024-01-27T23:21:42.272 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /home 2024-01-27T23:21:42.272 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /tmp 2024-01-27T23:21:42.272 INFO:teuthology.orchestra.run.smithi121.stdout:skipping the directory /dev 2024-01-27T23:21:42.272 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-selinux-2:16.2.4-0.el8.x86_64 26/29 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /sys 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /proc 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /run 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /mnt 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /var/tmp 2024-01-27T23:21:43.577 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /home 2024-01-27T23:21:43.578 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /tmp 2024-01-27T23:21:43.578 INFO:teuthology.orchestra.run.smithi194.stdout:skipping the directory /dev 2024-01-27T23:21:43.578 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:43.654 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-27T23:21:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-27T23:21:43.727 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-27T23:21:43.988 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-27T23:21:43.989 INFO:teuthology.orchestra.run.smithi121.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-27T23:21:44.735 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-27T23:21:44.735 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-27T23:21:44.735 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-27T23:21:44.735 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-27T23:21:44.735 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-27T23:21:44.736 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-27T23:21:44.736 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-27T23:21:44.736 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-27T23:21:44.736 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-27T23:21:44.736 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-27T23:21:44.736 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-27T23:21:44.737 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-27T23:21:44.737 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-27T23:21:44.737 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-27T23:21:44.737 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-27T23:21:44.737 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-27T23:21:44.737 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-27T23:21:44.737 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-27T23:21:44.738 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-27T23:21:44.739 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-27T23:21:44.739 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-27T23:21:44.739 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-27T23:21:44.909 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-27T23:21:44.970 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-radosgw-2:16.2.4-0.el8.x86_64 27/29 2024-01-27T23:21:44.970 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-27T23:21:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librbd1-1:12.2.7-9.el8.x86_64 28/29 2024-01-27T23:21:45.165 INFO:teuthology.orchestra.run.smithi194.stdout: Cleanup : librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-27T23:21:45.844 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados2-1:12.2.7-9.el8.x86_64 29/29 2024-01-27T23:21:45.844 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-base-2:16.2.4-0.el8.x86_64 1/29 2024-01-27T23:21:45.844 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-common-2:16.2.4-0.el8.x86_64 2/29 2024-01-27T23:21:45.844 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-radosgw-2:16.2.4-0.el8.x86_64 3/29 2024-01-27T23:21:45.844 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-selinux-2:16.2.4-0.el8.x86_64 4/29 2024-01-27T23:21:45.844 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs2-2:16.2.4-0.el8.x86_64 5/29 2024-01-27T23:21:45.844 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libradosstriper1-2:16.2.4-0.el8.x86_64 6/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librgw2-2:16.2.4-0.el8.x86_64 7/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-argparse-2:16.2.4-0.el8.x86_64 8/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-ceph-common-2:16.2.4-0.el8.x86_64 9/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cephfs-2:16.2.4-0.el8.x86_64 10/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rados-2:16.2.4-0.el8.x86_64 11/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rbd-2:16.2.4-0.el8.x86_64 12/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rgw-2:16.2.4-0.el8.x86_64 13/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libpmemobj-1.6.1-1.el8.x86_64 14/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librdkafka-1.6.1-1.el8.x86_64 15/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : lttng-ust-2.8.1-11.el8.x86_64 16/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-prettytable-0.7.2-14.el8.noarch 17/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : daxctl-libs-71.1-7.el8.x86_64 18/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librabbitmq-0.9.0-5.el8.x86_64 19/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ndctl-libs-71.1-7.el8.x86_64 20/29 2024-01-27T23:21:45.845 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fmt-6.2.1-1.el8.x86_64 21/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : gperftools-libs-1:2.7-9.el8.x86_64 22/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : leveldb-1.22-1.el8.x86_64 23/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : liboath-2.6.2-3.el8.x86_64 24/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libunwind-1.3.1-3.el8.x86_64 25/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-2:16.2.4-0.el8.x86_64 26/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados2-1:12.2.7-9.el8.x86_64 27/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-2:16.2.4-0.el8.x86_64 28/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi121.stdout:Upgraded: 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi121.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.846 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: leveldb-1.22-1.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.847 INFO:teuthology.orchestra.run.smithi121.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:45.848 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:21:46.116 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-test 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librbd1-1:12.2.7-9.el8.x86_64 29/29 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout:Upgraded: 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: librados2-2:16.2.4-0.el8.x86_64 librbd1-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-base-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-common-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-radosgw-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-selinux-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: daxctl-libs-71.1-7.el8.x86_64 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: fmt-6.2.1-1.el8.x86_64 2024-01-27T23:21:46.785 INFO:teuthology.orchestra.run.smithi194.stdout: gperftools-libs-1:2.7-9.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: leveldb-1.22-1.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs2-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: liboath-2.6.2-3.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: libpmemobj-1.6.1-1.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: librabbitmq-0.9.0-5.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: libradosstriper1-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: librdkafka-1.6.1-1.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: librgw2-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: libunwind-1.3.1-3.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: lttng-ust-2.8.1-11.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: ndctl-libs-71.1-7.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-argparse-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: python3-ceph-common-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cephfs-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: python3-prettytable-0.7.2-14.el8.noarch 2024-01-27T23:21:46.786 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rados-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.787 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rbd-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.787 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rgw-2:16.2.4-0.el8.x86_64 2024-01-27T23:21:46.787 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:46.787 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:21:46.871 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:43 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:21:47.063 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-test 2024-01-27T23:21:47.823 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:43 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:21:48.364 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:21:48.365 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-27T23:21:48.366 INFO:teuthology.orchestra.run.smithi121.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-27T23:21:48.367 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:48.367 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:21:48.367 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:21:48.367 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-01-27T23:21:48.367 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:48.367 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 53 M 2024-01-27T23:21:48.368 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 229 M 2024-01-27T23:21:48.368 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:21:48.678 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 3.1 MB/s | 169 kB 00:00 2024-01-27T23:21:49.311 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:21:49.313 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:21:49.313 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:21:49.313 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:21:49.313 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:21:49.313 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test x86_64 2:16.2.4-0.el8 ceph 52 M 2024-01-27T23:21:49.313 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:21:49.313 INFO:teuthology.orchestra.run.smithi194.stdout: jq x86_64 1.6-8.el8 appstream 203 k 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite x86_64 2:16.2.4-0.el8 ceph 169 k 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout: oniguruma x86_64 6.8.2-3.el8 appstream 188 k 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout: socat x86_64 1.7.4.1-1.el8 appstream 323 k 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout: xmlstarlet x86_64 1.6.1-20.el8 appstream 71 k 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 53 M 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 229 M 2024-01-27T23:21:49.314 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:21:49.370 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 70 MB/s | 52 MB 00:00 2024-01-27T23:21:49.429 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 252 kB/s | 203 kB 00:00 2024-01-27T23:21:49.454 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 243 kB/s | 188 kB 00:00 2024-01-27T23:21:49.496 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): libcephsqlite-16.2.4-0.el8.x86_64.rpm 2.7 MB/s | 169 kB 00:00 2024-01-27T23:21:49.571 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 605 kB/s | 71 kB 00:00 2024-01-27T23:21:49.813 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 731 kB/s | 323 kB 00:00 2024-01-27T23:21:49.814 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:21:49.814 INFO:teuthology.orchestra.run.smithi121.stdout:Total 37 MB/s | 53 MB 00:01 2024-01-27T23:21:49.857 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:21:49.878 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:21:49.878 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:21:50.137 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): ceph-test-16.2.4-0.el8.x86_64.rpm 74 MB/s | 52 MB 00:00 2024-01-27T23:21:50.163 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): jq-1.6-8.el8.x86_64.rpm 279 kB/s | 203 kB 00:00 2024-01-27T23:21:50.188 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): oniguruma-6.8.2-3.el8.x86_64.rpm 272 kB/s | 188 kB 00:00 2024-01-27T23:21:50.199 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:21:50.202 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:21:50.255 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): xmlstarlet-1.6.1-20.el8.x86_64.rpm 769 kB/s | 71 kB 00:00 2024-01-27T23:21:50.563 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): socat-1.7.4.1-1.el8.x86_64.rpm 760 kB/s | 323 kB 00:00 2024-01-27T23:21:50.564 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:21:50.565 INFO:teuthology.orchestra.run.smithi194.stdout:Total 42 MB/s | 53 MB 00:01 2024-01-27T23:21:50.607 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:21:50.628 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:21:50.628 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:21:50.682 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:21:50.912 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-27T23:21:50.938 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:21:50.941 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:21:51.113 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-27T23:21:51.293 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-27T23:21:51.367 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-27T23:21:51.451 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:21:51.542 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-27T23:21:51.656 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : xmlstarlet-1.6.1-20.el8.x86_64 1/6 2024-01-27T23:21:51.726 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-27T23:21:51.874 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : socat-1.7.4.1-1.el8.x86_64 2/6 2024-01-27T23:21:52.012 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-27T23:21:52.084 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: oniguruma-6.8.2-3.el8.x86_64 3/6 2024-01-27T23:21:52.277 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : jq-1.6-8.el8.x86_64 4/6 2024-01-27T23:21:52.437 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-27T23:21:56.558 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-27T23:21:56.807 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-27T23:21:57.099 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-27T23:21:57.099 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-27T23:21:57.099 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-27T23:21:57.099 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-27T23:21:57.099 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-27T23:21:57.099 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-27T23:21:57.284 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephsqlite-2:16.2.4-0.el8.x86_64 5/6 2024-01-27T23:21:57.523 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-27T23:21:57.524 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:57.524 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:21:57.524 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-27T23:21:57.524 INFO:teuthology.orchestra.run.smithi121.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-27T23:21:57.524 INFO:teuthology.orchestra.run.smithi121.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-27T23:21:57.524 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:21:57.524 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:21:57.526 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-27T23:21:57.751 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph 2024-01-27T23:21:57.800 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-test-2:16.2.4-0.el8.x86_64 6/6 2024-01-27T23:21:57.800 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-test-2:16.2.4-0.el8.x86_64 1/6 2024-01-27T23:21:57.800 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephsqlite-2:16.2.4-0.el8.x86_64 2/6 2024-01-27T23:21:57.800 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : jq-1.6-8.el8.x86_64 3/6 2024-01-27T23:21:57.800 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : oniguruma-6.8.2-3.el8.x86_64 4/6 2024-01-27T23:21:57.800 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : socat-1.7.4.1-1.el8.x86_64 5/6 2024-01-27T23:21:58.221 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : xmlstarlet-1.6.1-20.el8.x86_64 6/6 2024-01-27T23:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-test-2:16.2.4-0.el8.x86_64 jq-1.6-8.el8.x86_64 2024-01-27T23:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: libcephsqlite-2:16.2.4-0.el8.x86_64 oniguruma-6.8.2-3.el8.x86_64 2024-01-27T23:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: socat-1.7.4.1-1.el8.x86_64 xmlstarlet-1.6.1-20.el8.x86_64 2024-01-27T23:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:21:58.222 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:21:58.438 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph 2024-01-27T23:21:58.506 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:00:55 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:21:59.194 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:00:55 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:00.009 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:00.012 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:00.012 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-27T23:22:00.013 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-27T23:22:00.014 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout:Install 32 Packages 2024-01-27T23:22:00.015 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:00.016 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 31 M 2024-01-27T23:22:00.016 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 111 M 2024-01-27T23:22:00.016 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:22:00.499 INFO:teuthology.orchestra.run.smithi121.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 103 kB/s | 6.3 kB 00:00 2024-01-27T23:22:00.549 INFO:teuthology.orchestra.run.smithi121.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 14 MB/s | 1.6 MB 00:00 2024-01-27T23:22:00.608 INFO:teuthology.orchestra.run.smithi121.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 14 MB/s | 2.4 MB 00:00 2024-01-27T23:22:00.633 INFO:teuthology.orchestra.run.smithi121.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.6 MB/s | 220 kB 00:00 2024-01-27T23:22:00.703 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:00.706 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-27T23:22:00.706 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:00.706 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:22:00.706 INFO:teuthology.orchestra.run.smithi194.stdout: ceph x86_64 2:16.2.4-0.el8 ceph 6.3 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds x86_64 2:16.2.4-0.el8 ceph 2.4 M 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr x86_64 2:16.2.4-0.el8 ceph 1.6 M 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core noarch 2:16.2.4-0.el8 ceph-noarch 220 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon x86_64 2:16.2.4-0.el8 ceph 5.0 M 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd x86_64 2:16.2.4-0.el8 ceph 18 M 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt x86_64 3.1.6-2.el8.1 epel 44 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4 noarch 4.6.3-2.el8.1 epel 185 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi x86_64 1.11.5-6.el8 baseos 249 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot noarch 8.5.2-1.el8 epel 173 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy noarch 18.4.0-1.el8 epel 384 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography x86_64 3.2.1-7.el8 baseos 641 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco noarch 6.2-6.el8 epel 11 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools noarch 2.0-4.el8 epel 18 k 2024-01-27T23:22:00.707 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils noarch 0.3.5-11.el8 epel 49 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako noarch 1.0.6-14.el8 appstream 157 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe x86_64 0.23-19.el8 appstream 39 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools noarch 7.2.0-3.el8 epel 59 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan noarch 1.3.2-9.el8 epel 283 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend noarch 2.6-1.el8 epel 16 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL noarch 19.0.0-1.el8 appstream 103 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser noarch 2.14-14.el8 baseos 109 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz noarch 2017.2-11.el8 appstream 57 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric noarch 0.8.1-17.el8 epel 19 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch noarch 3.4.0.3-18.el8 epel 24 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora noarch 1.14.1-5.el8 epel 29 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme noarch 0.6.0-4.el8 epel 27 k 2024-01-27T23:22:00.708 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress noarch 1.4.3-1.el8 epel 255 k 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob noarch 1.8.5-1.el8.1 epel 251 k 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest noarch 2.0.33-1.el8 epel 85 k 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug noarch 0.12.2-4.el8 appstream 457 k 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile noarch 2.0-2.el8 epel 23 k 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout:Install 32 Packages 2024-01-27T23:22:00.709 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:00.710 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 31 M 2024-01-27T23:22:00.710 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 111 M 2024-01-27T23:22:00.710 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:22:00.737 INFO:teuthology.orchestra.run.smithi121.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 21 MB/s | 5.0 MB 00:00 2024-01-27T23:22:00.967 INFO:teuthology.orchestra.run.smithi121.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-27T23:22:00.972 INFO:teuthology.orchestra.run.smithi194.stdout:(1/32): ceph-16.2.4-0.el8.x86_64.rpm 123 kB/s | 6.3 kB 00:00 2024-01-27T23:22:01.015 INFO:teuthology.orchestra.run.smithi194.stdout:(2/32): ceph-mgr-16.2.4-0.el8.x86_64.rpm 17 MB/s | 1.6 MB 00:00 2024-01-27T23:22:01.026 INFO:teuthology.orchestra.run.smithi121.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 400 kB/s | 157 kB 00:00 2024-01-27T23:22:01.073 INFO:teuthology.orchestra.run.smithi194.stdout:(3/32): ceph-mds-16.2.4-0.el8.x86_64.rpm 16 MB/s | 2.4 MB 00:00 2024-01-27T23:22:01.098 INFO:teuthology.orchestra.run.smithi194.stdout:(4/32): ceph-mgr-modules-core-16.2.4-0.el8.noar 8.7 MB/s | 220 kB 00:00 2024-01-27T23:22:01.127 INFO:teuthology.orchestra.run.smithi121.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 571 kB/s | 57 kB 00:00 2024-01-27T23:22:01.152 INFO:teuthology.orchestra.run.smithi121.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 94 kB/s | 39 kB 00:00 2024-01-27T23:22:01.165 INFO:teuthology.orchestra.run.smithi194.stdout:(5/32): ceph-mon-16.2.4-0.el8.x86_64.rpm 26 MB/s | 5.0 MB 00:00 2024-01-27T23:22:01.294 INFO:teuthology.orchestra.run.smithi121.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 317 kB/s | 103 kB 00:00 2024-01-27T23:22:01.327 INFO:teuthology.orchestra.run.smithi121.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 2.2 MB/s | 457 kB 00:00 2024-01-27T23:22:01.424 INFO:teuthology.orchestra.run.smithi194.stdout:(6/32): ceph-osd-16.2.4-0.el8.x86_64.rpm 44 MB/s | 18 MB 00:00 2024-01-27T23:22:01.458 INFO:teuthology.orchestra.run.smithi194.stdout:(7/32): python3-mako-1.0.6-14.el8.noarch.rpm 437 kB/s | 157 kB 00:00 2024-01-27T23:22:01.558 INFO:teuthology.orchestra.run.smithi194.stdout:(8/32): python3-pytz-2017.2-11.el8.noarch.rpm 570 kB/s | 57 kB 00:00 2024-01-27T23:22:01.592 INFO:teuthology.orchestra.run.smithi194.stdout:(9/32): python3-markupsafe-0.23-19.el8.x86_64.r 91 kB/s | 39 kB 00:00 2024-01-27T23:22:01.594 INFO:teuthology.orchestra.run.smithi121.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 563 kB/s | 249 kB 00:00 2024-01-27T23:22:01.694 INFO:teuthology.orchestra.run.smithi121.stdout:(13/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 436 kB/s | 44 kB 00:00 2024-01-27T23:22:01.720 INFO:teuthology.orchestra.run.smithi121.stdout:(14/32): python3-pycparser-2.14-14.el8.noarch.r 278 kB/s | 109 kB 00:00 2024-01-27T23:22:01.767 INFO:teuthology.orchestra.run.smithi194.stdout:(10/32): python3-pyOpenSSL-19.0.0-1.el8.noarch. 302 kB/s | 103 kB 00:00 2024-01-27T23:22:01.770 INFO:teuthology.orchestra.run.smithi121.stdout:(15/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 2.4 MB/s | 185 kB 00:00 2024-01-27T23:22:01.792 INFO:teuthology.orchestra.run.smithi194.stdout:(11/32): python3-werkzeug-0.12.2-4.el8.noarch.r 1.9 MB/s | 457 kB 00:00 2024-01-27T23:22:01.854 INFO:teuthology.orchestra.run.smithi121.stdout:(16/32): python3-cherrypy-18.4.0-1.el8.noarch.r 4.5 MB/s | 384 kB 00:00 2024-01-27T23:22:01.879 INFO:teuthology.orchestra.run.smithi121.stdout:(17/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 1.1 MB/s | 173 kB 00:00 2024-01-27T23:22:01.896 INFO:teuthology.orchestra.run.smithi121.stdout:(18/32): python3-jaraco-6.2-6.el8.noarch.rpm 265 kB/s | 11 kB 00:00 2024-01-27T23:22:01.913 INFO:teuthology.orchestra.run.smithi121.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 523 kB/s | 18 kB 00:00 2024-01-27T23:22:01.929 INFO:teuthology.orchestra.run.smithi121.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 1.4 MB/s | 49 kB 00:00 2024-01-27T23:22:02.042 INFO:teuthology.orchestra.run.smithi121.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 1.7 MB/s | 59 kB 00:00 2024-01-27T23:22:02.043 INFO:teuthology.orchestra.run.smithi121.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 468 kB/s | 16 kB 00:00 2024-01-27T23:22:02.043 INFO:teuthology.orchestra.run.smithi121.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 3.7 MB/s | 283 kB 00:00 2024-01-27T23:22:02.043 INFO:teuthology.orchestra.run.smithi121.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 447 kB/s | 19 kB 00:00 2024-01-27T23:22:02.043 INFO:teuthology.orchestra.run.smithi194.stdout:(12/32): python3-cffi-1.11.5-6.el8.x86_64.rpm 552 kB/s | 249 kB 00:00 2024-01-27T23:22:02.048 INFO:teuthology.orchestra.run.smithi121.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 573 kB/s | 24 kB 00:00 2024-01-27T23:22:02.065 INFO:teuthology.orchestra.run.smithi121.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 698 kB/s | 29 kB 00:00 2024-01-27T23:22:02.098 INFO:teuthology.orchestra.run.smithi121.stdout:(27/32): python3-cryptography-3.2.1-7.el8.x86_6 797 kB/s | 641 kB 00:00 2024-01-27T23:22:02.124 INFO:teuthology.orchestra.run.smithi121.stdout:(28/32): python3-trustme-0.6.0-4.el8.noarch.rpm 364 kB/s | 27 kB 00:00 2024-01-27T23:22:02.149 INFO:teuthology.orchestra.run.smithi121.stdout:(29/32): python3-waitress-1.4.3-1.el8.noarch.rp 3.0 MB/s | 255 kB 00:00 2024-01-27T23:22:02.199 INFO:teuthology.orchestra.run.smithi121.stdout:(30/32): python3-webtest-2.0.33-1.el8.noarch.rp 1.1 MB/s | 85 kB 00:00 2024-01-27T23:22:02.224 INFO:teuthology.orchestra.run.smithi121.stdout:(31/32): python3-zc-lockfile-2.0-2.el8.noarch.r 306 kB/s | 23 kB 00:00 2024-01-27T23:22:02.266 INFO:teuthology.orchestra.run.smithi121.stdout:(32/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 1.5 MB/s | 251 kB 00:00 2024-01-27T23:22:02.267 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:02.268 INFO:teuthology.orchestra.run.smithi121.stdout:Total 14 MB/s | 31 MB 00:02 2024-01-27T23:22:02.349 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:22:02.388 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:22:02.388 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:22:02.476 INFO:teuthology.orchestra.run.smithi194.stdout:(13/32): python3-pycparser-2.14-14.el8.noarch.r 159 kB/s | 109 kB 00:00 2024-01-27T23:22:02.610 INFO:teuthology.orchestra.run.smithi194.stdout:(14/32): python3-bcrypt-3.1.6-2.el8.1.x86_64.rp 77 kB/s | 44 kB 00:00 2024-01-27T23:22:02.711 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:22:02.714 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:22:02.868 INFO:teuthology.orchestra.run.smithi194.stdout:(15/32): python3-cryptography-3.2.1-7.el8.x86_6 582 kB/s | 641 kB 00:01 2024-01-27T23:22:02.894 INFO:teuthology.orchestra.run.smithi194.stdout:(16/32): python3-cheroot-8.5.2-1.el8.noarch.rpm 609 kB/s | 173 kB 00:00 2024-01-27T23:22:02.994 INFO:teuthology.orchestra.run.smithi194.stdout:(17/32): python3-jaraco-6.2-6.el8.noarch.rpm 111 kB/s | 11 kB 00:00 2024-01-27T23:22:03.027 INFO:teuthology.orchestra.run.smithi194.stdout:(18/32): python3-beautifulsoup4-4.6.3-2.el8.1.n 336 kB/s | 185 kB 00:00 2024-01-27T23:22:03.103 INFO:teuthology.orchestra.run.smithi194.stdout:(19/32): python3-jaraco-functools-2.0-4.el8.noa 162 kB/s | 18 kB 00:00 2024-01-27T23:22:03.144 INFO:teuthology.orchestra.run.smithi194.stdout:(20/32): python3-logutils-0.3.5-11.el8.noarch.r 417 kB/s | 49 kB 00:00 2024-01-27T23:22:03.228 INFO:teuthology.orchestra.run.smithi194.stdout:(21/32): python3-more-itertools-7.2.0-3.el8.noa 469 kB/s | 59 kB 00:00 2024-01-27T23:22:03.337 INFO:teuthology.orchestra.run.smithi194.stdout:(22/32): python3-portend-2.6-1.el8.noarch.rpm 144 kB/s | 16 kB 00:00 2024-01-27T23:22:03.362 INFO:teuthology.orchestra.run.smithi194.stdout:(23/32): python3-pecan-1.3.2-9.el8.noarch.rpm 1.3 MB/s | 283 kB 00:00 2024-01-27T23:22:03.446 INFO:teuthology.orchestra.run.smithi194.stdout:(24/32): python3-simplegeneric-0.8.1-17.el8.noa 173 kB/s | 19 kB 00:00 2024-01-27T23:22:03.488 INFO:teuthology.orchestra.run.smithi194.stdout:(25/32): python3-singledispatch-3.4.0.3-18.el8. 191 kB/s | 24 kB 00:00 2024-01-27T23:22:03.546 INFO:teuthology.orchestra.run.smithi194.stdout:(26/32): python3-tempora-1.14.1-5.el8.noarch.rp 292 kB/s | 29 kB 00:00 2024-01-27T23:22:03.588 INFO:teuthology.orchestra.run.smithi194.stdout:(27/32): python3-trustme-0.6.0-4.el8.noarch.rpm 275 kB/s | 27 kB 00:00 2024-01-27T23:22:03.613 INFO:teuthology.orchestra.run.smithi194.stdout:(28/32): python3-cherrypy-18.4.0-1.el8.noarch.r 516 kB/s | 384 kB 00:00 2024-01-27T23:22:03.714 INFO:teuthology.orchestra.run.smithi194.stdout:(29/32): python3-webob-1.8.5-1.el8.1.noarch.rpm 2.0 MB/s | 251 kB 00:00 2024-01-27T23:22:03.739 INFO:teuthology.orchestra.run.smithi194.stdout:(30/32): python3-waitress-1.4.3-1.el8.noarch.rp 1.3 MB/s | 255 kB 00:00 2024-01-27T23:22:03.764 INFO:teuthology.orchestra.run.smithi194.stdout:(31/32): python3-webtest-2.0.33-1.el8.noarch.rp 567 kB/s | 85 kB 00:00 2024-01-27T23:22:03.806 INFO:teuthology.orchestra.run.smithi194.stdout:(32/32): python3-zc-lockfile-2.0-2.el8.noarch.r 249 kB/s | 23 kB 00:00 2024-01-27T23:22:03.810 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:03.810 INFO:teuthology.orchestra.run.smithi194.stdout:Total 10 MB/s | 31 MB 00:03 2024-01-27T23:22:03.872 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:22:03.891 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:22:03.929 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:22:03.929 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:22:04.087 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-27T23:22:04.245 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:22:04.248 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:22:04.265 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-27T23:22:04.530 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-27T23:22:04.763 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-27T23:22:04.944 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-27T23:22:05.141 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-27T23:22:05.318 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-27T23:22:05.415 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:22:05.512 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-27T23:22:05.665 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-more-itertools-7.2.0-3.el8.noarch 1/32 2024-01-27T23:22:05.696 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-27T23:22:05.892 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webob-1.8.5-1.el8.1.noarch 2/32 2024-01-27T23:22:05.917 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-27T23:22:06.096 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-27T23:22:06.159 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-zc-lockfile-2.0-2.el8.noarch 3/32 2024-01-27T23:22:06.388 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-27T23:22:06.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-waitress-1.4.3-1.el8.noarch 4/32 2024-01-27T23:22:06.588 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-singledispatch-3.4.0.3-18.el8.noarch 5/32 2024-01-27T23:22:06.768 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-simplegeneric-0.8.1-17.el8.noarch 6/32 2024-01-27T23:22:06.776 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-27T23:22:06.962 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-logutils-0.3.5-11.el8.noarch 7/32 2024-01-27T23:22:07.059 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-27T23:22:07.140 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-6.2-6.el8.noarch 8/32 2024-01-27T23:22:07.239 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-27T23:22:07.365 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jaraco-functools-2.0-4.el8.noarch 9/32 2024-01-27T23:22:07.522 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-27T23:22:07.569 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 10/32 2024-01-27T23:22:07.740 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-27T23:22:07.816 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-webtest-2.0.33-1.el8.noarch 11/32 2024-01-27T23:22:08.023 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-27T23:22:08.056 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pycparser-2.14-14.el8.noarch 12/32 2024-01-27T23:22:08.253 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-27T23:22:08.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cffi-1.11.5-6.el8.x86_64 13/32 2024-01-27T23:22:08.458 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-27T23:22:08.636 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-27T23:22:08.695 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cryptography-3.2.1-7.el8.x86_64 14/32 2024-01-27T23:22:08.850 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyOpenSSL-19.0.0-1.el8.noarch 15/32 2024-01-27T23:22:08.892 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-27T23:22:09.133 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-trustme-0.6.0-4.el8.noarch 16/32 2024-01-27T23:22:09.163 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-27T23:22:09.343 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-27T23:22:09.408 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-27T23:22:09.634 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-bcrypt-3.1.6-2.el8.1.x86_64 18/32 2024-01-27T23:22:09.763 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-27T23:22:09.872 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-werkzeug-0.12.2-4.el8.noarch 19/32 2024-01-27T23:22:10.052 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pytz-2017.2-11.el8.noarch 20/32 2024-01-27T23:22:10.079 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-27T23:22:10.272 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-tempora-1.14.1-5.el8.noarch 21/32 2024-01-27T23:22:10.493 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-27T23:22:10.531 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-portend-2.6-1.el8.noarch 22/32 2024-01-27T23:22:10.731 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-27T23:22:10.792 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cherrypy-18.4.0-1.el8.noarch 23/32 2024-01-27T23:22:11.084 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-markupsafe-0.23-19.el8.x86_64 24/32 2024-01-27T23:22:11.429 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-mako-1.0.6-14.el8.noarch 25/32 2024-01-27T23:22:11.748 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pecan-1.3.2-9.el8.noarch 26/32 2024-01-27T23:22:12.150 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 27/32 2024-01-27T23:22:12.273 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-27T23:22:12.380 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-27T23:22:12.481 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-27T23:22:12.913 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-27T23:22:13.131 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-27T23:22:13.353 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-27T23:22:13.572 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-27T23:22:13.590 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-27T23:22:13.721 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-27T23:22:13.911 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-2:16.2.4-0.el8.x86_64 28/32 2024-01-27T23:22:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-27T23:22:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-27T23:22:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-27T23:22:14.120 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-27T23:22:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-27T23:22:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-27T23:22:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-27T23:22:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-27T23:22:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-27T23:22:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-27T23:22:14.121 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-27T23:22:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-27T23:22:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-27T23:22:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-27T23:22:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-27T23:22:14.122 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-27T23:22:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-27T23:22:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-27T23:22:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-27T23:22:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-27T23:22:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-27T23:22:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-27T23:22:14.123 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-27T23:22:14.124 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-27T23:22:14.129 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-27T23:22:14.561 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-osd-2:16.2.4-0.el8.x86_64 29/32 2024-01-27T23:22:14.812 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-27T23:22:14.998 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-27T23:22:14.998 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-27T23:22:14.999 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-portend-2.6-1.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-27T23:22:15.000 INFO:teuthology.orchestra.run.smithi121.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:15.001 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:15.034 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mon-2:16.2.4-0.el8.x86_64 30/32 2024-01-27T23:22:15.242 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-27T23:22:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mds-2:16.2.4-0.el8.x86_64 31/32 2024-01-27T23:22:15.268 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-base 2024-01-27T23:22:15.408 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-27T23:22:15.815 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-2:16.2.4-0.el8.x86_64 32/32 2024-01-27T23:22:15.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-2:16.2.4-0.el8.x86_64 1/32 2024-01-27T23:22:15.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mds-2:16.2.4-0.el8.x86_64 2/32 2024-01-27T23:22:15.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-2:16.2.4-0.el8.x86_64 3/32 2024-01-27T23:22:15.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mon-2:16.2.4-0.el8.x86_64 4/32 2024-01-27T23:22:15.815 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-osd-2:16.2.4-0.el8.x86_64 5/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 6/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-mako-1.0.6-14.el8.noarch 7/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-markupsafe-0.23-19.el8.x86_64 8/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyOpenSSL-19.0.0-1.el8.noarch 9/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pytz-2017.2-11.el8.noarch 10/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-werkzeug-0.12.2-4.el8.noarch 11/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cffi-1.11.5-6.el8.x86_64 12/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cryptography-3.2.1-7.el8.x86_64 13/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pycparser-2.14-14.el8.noarch 14/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-bcrypt-3.1.6-2.el8.1.x86_64 15/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-beautifulsoup4-4.6.3-2.el8.1.noarch 16/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cheroot-8.5.2-1.el8.noarch 17/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cherrypy-18.4.0-1.el8.noarch 18/32 2024-01-27T23:22:15.816 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-6.2-6.el8.noarch 19/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jaraco-functools-2.0-4.el8.noarch 20/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-logutils-0.3.5-11.el8.noarch 21/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-more-itertools-7.2.0-3.el8.noarch 22/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pecan-1.3.2-9.el8.noarch 23/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-portend-2.6-1.el8.noarch 24/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-simplegeneric-0.8.1-17.el8.noarch 25/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-singledispatch-3.4.0.3-18.el8.noarch 26/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-tempora-1.14.1-5.el8.noarch 27/32 2024-01-27T23:22:15.817 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-trustme-0.6.0-4.el8.noarch 28/32 2024-01-27T23:22:15.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-waitress-1.4.3-1.el8.noarch 29/32 2024-01-27T23:22:15.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webob-1.8.5-1.el8.1.noarch 30/32 2024-01-27T23:22:15.818 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-webtest-2.0.33-1.el8.noarch 31/32 2024-01-27T23:22:16.020 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:13 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:16.635 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-zc-lockfile-2.0-2.el8.noarch 32/32 2024-01-27T23:22:16.635 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mds-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-modules-core-2:16.2.4-0.el8.noarch 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mon-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-osd-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: python3-bcrypt-3.1.6-2.el8.1.x86_64 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: python3-beautifulsoup4-4.6.3-2.el8.1.noarch 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cffi-1.11.5-6.el8.x86_64 2024-01-27T23:22:16.636 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cheroot-8.5.2-1.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cherrypy-18.4.0-1.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cryptography-3.2.1-7.el8.x86_64 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-6.2-6.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jaraco-functools-2.0-4.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-logutils-0.3.5-11.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-mako-1.0.6-14.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-markupsafe-0.23-19.el8.x86_64 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-more-itertools-7.2.0-3.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pecan-1.3.2-9.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-portend-2.6-1.el8.noarch 2024-01-27T23:22:16.637 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyOpenSSL-19.0.0-1.el8.noarch 2024-01-27T23:22:16.638 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pycparser-2.14-14.el8.noarch 2024-01-27T23:22:16.638 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pytz-2017.2-11.el8.noarch 2024-01-27T23:22:16.638 INFO:teuthology.orchestra.run.smithi194.stdout: python3-simplegeneric-0.8.1-17.el8.noarch 2024-01-27T23:22:16.638 INFO:teuthology.orchestra.run.smithi194.stdout: python3-singledispatch-3.4.0.3-18.el8.noarch 2024-01-27T23:22:16.638 INFO:teuthology.orchestra.run.smithi194.stdout: python3-tempora-1.14.1-5.el8.noarch 2024-01-27T23:22:16.638 INFO:teuthology.orchestra.run.smithi194.stdout: python3-trustme-0.6.0-4.el8.noarch 2024-01-27T23:22:16.639 INFO:teuthology.orchestra.run.smithi194.stdout: python3-waitress-1.4.3-1.el8.noarch 2024-01-27T23:22:16.639 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webob-1.8.5-1.el8.1.noarch 2024-01-27T23:22:16.639 INFO:teuthology.orchestra.run.smithi194.stdout: python3-webtest-2.0.33-1.el8.noarch 2024-01-27T23:22:16.639 INFO:teuthology.orchestra.run.smithi194.stdout: python3-werkzeug-0.12.2-4.el8.noarch 2024-01-27T23:22:16.639 INFO:teuthology.orchestra.run.smithi194.stdout: python3-zc-lockfile-2.0-2.el8.noarch 2024-01-27T23:22:16.640 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:16.640 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:16.921 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-base 2024-01-27T23:22:17.473 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:22:17.519 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:17.520 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:22:17.520 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:17.624 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install cephadm 2024-01-27T23:22:17.676 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:13 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:18.363 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:15 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:19.113 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-base-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:22:19.159 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:19.160 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:22:19.160 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:19.265 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install cephadm 2024-01-27T23:22:19.858 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:19.859 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-01-27T23:22:19.860 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:19.860 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 71 k 2024-01-27T23:22:19.860 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 290 k 2024-01-27T23:22:19.860 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:22:19.906 INFO:teuthology.orchestra.run.smithi121.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.5 MB/s | 71 kB 00:00 2024-01-27T23:22:19.906 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:19.906 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.5 MB/s | 71 kB 00:00 2024-01-27T23:22:19.907 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:22:19.913 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:22:19.913 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:22:19.915 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:22:19.918 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:22:19.965 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:22:20.010 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:16 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:20.474 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:20.667 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:20.895 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:21.297 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:21.297 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:21.297 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:22:21.297 INFO:teuthology.orchestra.run.smithi121.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-27T23:22:21.297 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:21.298 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:21.515 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-immutable-object-cache 2024-01-27T23:22:21.515 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm noarch 2:16.2.4-0.el8 ceph-noarch 71 k 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 71 k 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 290 k 2024-01-27T23:22:21.517 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:22:21.560 INFO:teuthology.orchestra.run.smithi194.stdout:cephadm-16.2.4-0.el8.noarch.rpm 1.6 MB/s | 71 kB 00:00 2024-01-27T23:22:21.561 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:21.561 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.6 MB/s | 71 kB 00:00 2024-01-27T23:22:21.561 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:22:21.568 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:22:21.568 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:22:21.570 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:22:21.573 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:22:21.620 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:22:22.099 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:22.271 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:19 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:22.290 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:22.523 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:22.898 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : cephadm-2:16.2.4-0.el8.noarch 1/1 2024-01-27T23:22:22.898 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:22.898 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:22:22.898 INFO:teuthology.orchestra.run.smithi194.stdout: cephadm-2:16.2.4-0.el8.noarch 2024-01-27T23:22:22.899 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:22.899 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:23.107 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-immutable-object-cache 2024-01-27T23:22:23.762 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repo Size 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-01-27T23:22:23.763 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:23.764 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 163 k 2024-01-27T23:22:23.764 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 587 k 2024-01-27T23:22:23.764 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:22:23.811 INFO:teuthology.orchestra.run.smithi121.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.4 MB/s | 163 kB 00:00 2024-01-27T23:22:23.811 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:23.812 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.3 MB/s | 163 kB 00:00 2024-01-27T23:22:23.812 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:22:23.822 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:22:23.822 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:22:23.867 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:19 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:23.868 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:22:23.872 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:22:24.003 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:22:24.169 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:22:24.587 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:22:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:22:24.992 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:24.993 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:22:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:24.993 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:24.993 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:25.204 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr 2024-01-27T23:22:25.363 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repo Size 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache x86_64 2:16.2.4-0.el8 ceph 163 k 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-27T23:22:25.364 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:25.365 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 163 k 2024-01-27T23:22:25.365 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 587 k 2024-01-27T23:22:25.365 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:22:25.415 INFO:teuthology.orchestra.run.smithi194.stdout:ceph-immutable-object-cache-16.2.4-0.el8.x86_64 3.2 MB/s | 163 kB 00:00 2024-01-27T23:22:25.415 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:25.415 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.2 MB/s | 163 kB 00:00 2024-01-27T23:22:25.416 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:22:25.427 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:22:25.427 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:22:25.474 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:22:25.476 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:22:25.634 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:22:25.839 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:22:25.958 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:22 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:26.258 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:22:26.652 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:22:26.652 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:26.652 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:22:26.652 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-immutable-object-cache-2:16.2.4-0.el8.x86_64 2024-01-27T23:22:26.652 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:26.652 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:26.886 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr 2024-01-27T23:22:27.405 INFO:teuthology.orchestra.run.smithi121.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:22:27.451 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:27.452 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:22:27.452 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:27.547 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-dashboard 2024-01-27T23:22:27.651 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:23 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:28.308 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:25 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:29.103 INFO:teuthology.orchestra.run.smithi194.stdout:Package ceph-mgr-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:22:29.149 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:29.150 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:22:29.150 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:29.244 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-dashboard 2024-01-27T23:22:29.788 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-27T23:22:29.790 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:22:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-01-27T23:22:29.791 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.0 M 2024-01-27T23:22:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 15 M 2024-01-27T23:22:29.791 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:22:29.994 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:26 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:30.031 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 414 kB/s | 24 kB 00:00 2024-01-27T23:22:30.057 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 107 kB/s | 8.9 kB 00:00 2024-01-27T23:22:30.132 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 17 MB/s | 2.7 MB 00:00 2024-01-27T23:22:30.241 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 183 kB/s | 34 kB 00:00 2024-01-27T23:22:30.299 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-routes-2.4.1-12.el8.noarch.rpm 1.2 MB/s | 196 kB 00:00 2024-01-27T23:22:30.366 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-jwt-1.6.1-2.el8.noarch.rpm 127 kB/s | 43 kB 00:00 2024-01-27T23:22:30.369 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:30.369 INFO:teuthology.orchestra.run.smithi121.stdout:Total 5.1 MB/s | 3.0 MB 00:00 2024-01-27T23:22:30.379 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:22:30.388 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:22:30.388 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:22:30.577 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:22:30.580 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:22:30.863 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:22:31.090 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-27T23:22:31.344 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-27T23:22:31.484 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:31.485 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:31.485 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard noarch 2:16.2.4-0.el8 ceph-noarch 2.7 M 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards noarch 2:16.2.4-0.el8 ceph-noarch 24 k 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts noarch 2:16.2.4-0.el8 ceph-noarch 8.9 k 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt noarch 1.6.1-2.el8 baseos 43 k 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru noarch 0.7-6.el8 epel 34 k 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes noarch 2.4.1-12.el8 epel 196 k 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-01-27T23:22:31.486 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:31.487 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.0 M 2024-01-27T23:22:31.487 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 15 M 2024-01-27T23:22:31.487 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:22:31.519 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-27T23:22:31.666 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-27T23:22:32.101 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): ceph-grafana-dashboards-16.2.4-0.el8.noa 484 kB/s | 24 kB 00:00 2024-01-27T23:22:32.126 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): ceph-prometheus-alerts-16.2.4-0.el8.noar 119 kB/s | 8.9 kB 00:00 2024-01-27T23:22:32.193 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): ceph-mgr-dashboard-16.2.4-0.el8.noarch.r 19 MB/s | 2.7 MB 00:00 2024-01-27T23:22:32.460 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-jwt-1.6.1-2.el8.noarch.rpm 119 kB/s | 43 kB 00:00 2024-01-27T23:22:32.535 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-repoze-lru-0.7-6.el8.noarch.rpm 82 kB/s | 34 kB 00:00 2024-01-27T23:22:32.639 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-27T23:22:32.802 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-routes-2.4.1-12.el8.noarch.rpm 323 kB/s | 196 kB 00:00 2024-01-27T23:22:32.805 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:32.806 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.2 MB/s | 3.0 MB 00:01 2024-01-27T23:22:32.817 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:22:32.827 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:22:32.827 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:22:32.909 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:22:33.000 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:22:33.003 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:22:33.198 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:22:33.198 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-27T23:22:33.198 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-27T23:22:33.198 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-27T23:22:33.198 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-27T23:22:33.198 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-27T23:22:33.305 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:22:33.536 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-repoze-lru-0.7-6.el8.noarch 1/6 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-27T23:22:33.596 INFO:teuthology.orchestra.run.smithi121.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-27T23:22:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-27T23:22:33.597 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:33.597 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:33.720 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-27T23:22:33.823 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-routes-2.4.1-12.el8.noarch 2/6 2024-01-27T23:22:34.048 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jwt-1.6.1-2.el8.noarch 3/6 2024-01-27T23:22:34.169 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 4/6 2024-01-27T23:22:34.463 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:31 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:35.132 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 5/6 2024-01-27T23:22:35.416 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:22:35.702 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:22:35.702 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 1/6 2024-01-27T23:22:35.702 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2/6 2024-01-27T23:22:35.702 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 3/6 2024-01-27T23:22:35.703 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jwt-1.6.1-2.el8.noarch 4/6 2024-01-27T23:22:35.703 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-repoze-lru-0.7-6.el8.noarch 5/6 2024-01-27T23:22:35.976 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:35.977 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-01-27T23:22:35.977 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-27T23:22:35.978 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:35.979 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:22:35.979 INFO:teuthology.orchestra.run.smithi121.stdout:============================================================================================ 2024-01-27T23:22:35.979 INFO:teuthology.orchestra.run.smithi121.stdout:Install 8 Packages 2024-01-27T23:22:35.979 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:35.979 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 37 M 2024-01-27T23:22:35.980 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 210 M 2024-01-27T23:22:35.980 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:22:36.089 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-routes-2.4.1-12.el8.noarch 6/6 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-grafana-dashboards-2:16.2.4-0.el8.noarch 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-dashboard-2:16.2.4-0.el8.noarch 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-prometheus-alerts-2:16.2.4-0.el8.noarch 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jwt-1.6.1-2.el8.noarch 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: python3-repoze-lru-0.7-6.el8.noarch 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: python3-routes-2.4.1-12.el8.noarch 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:36.090 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:36.203 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-diskprediction-local 2024-01-27T23:22:36.379 INFO:teuthology.orchestra.run.smithi121.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 49 MB/s | 8.6 MB 00:00 2024-01-27T23:22:36.956 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:32 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:36.996 INFO:teuthology.orchestra.run.smithi121.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 6.0 MB/s | 4.7 MB 00:00 2024-01-27T23:22:37.063 INFO:teuthology.orchestra.run.smithi121.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.7 MB/s | 4.9 MB 00:00 2024-01-27T23:22:37.122 INFO:teuthology.orchestra.run.smithi121.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 1.8 MB/s | 225 kB 00:00 2024-01-27T23:22:37.189 INFO:teuthology.orchestra.run.smithi121.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.6 MB/s | 3.7 MB 00:00 2024-01-27T23:22:37.706 INFO:teuthology.orchestra.run.smithi121.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 22 MB/s | 14 MB 00:00 2024-01-27T23:22:37.731 INFO:teuthology.orchestra.run.smithi121.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 318 kB/s | 172 kB 00:00 2024-01-27T23:22:38.073 INFO:teuthology.orchestra.run.smithi121.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 679 kB/s | 645 kB 00:00 2024-01-27T23:22:38.073 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:38.073 INFO:teuthology.orchestra.run.smithi121.stdout:Total 18 MB/s | 37 MB 00:02 2024-01-27T23:22:38.301 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:22:38.320 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:22:38.320 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:22:38.466 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:38.467 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local noarch 2:16.2.4-0.el8 ceph-noarch 8.6 M 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran x86_64 8.5.0-21.el8 baseos 645 k 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath x86_64 8.5.0-21.el8 baseos 172 k 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: openblas x86_64 0.3.15-4.el8 appstream 4.7 M 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads x86_64 0.3.15-4.el8 appstream 4.9 M 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy x86_64 1:1.14.3-10.el8 appstream 3.7 M 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py x86_64 1:1.14.3-10.el8 appstream 225 k 2024-01-27T23:22:38.468 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy x86_64 1.0.0-21.module_el8+762+77bd8591 appstream 14 M 2024-01-27T23:22:38.469 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:38.469 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:22:38.469 INFO:teuthology.orchestra.run.smithi194.stdout:============================================================================================ 2024-01-27T23:22:38.469 INFO:teuthology.orchestra.run.smithi194.stdout:Install 8 Packages 2024-01-27T23:22:38.469 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:38.469 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 37 M 2024-01-27T23:22:38.469 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 210 M 2024-01-27T23:22:38.470 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:22:38.684 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:22:38.687 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:22:39.005 INFO:teuthology.orchestra.run.smithi194.stdout:(1/8): ceph-mgr-diskprediction-local-16.2.4-0.e 51 MB/s | 8.6 MB 00:00 2024-01-27T23:22:39.325 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:22:39.491 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-27T23:22:39.622 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-27T23:22:39.631 INFO:teuthology.orchestra.run.smithi194.stdout:(2/8): openblas-0.3.15-4.el8.x86_64.rpm 5.9 MB/s | 4.7 MB 00:00 2024-01-27T23:22:39.715 INFO:teuthology.orchestra.run.smithi194.stdout:(3/8): openblas-threads-0.3.15-4.el8.x86_64.rpm 5.5 MB/s | 4.9 MB 00:00 2024-01-27T23:22:39.740 INFO:teuthology.orchestra.run.smithi194.stdout:(4/8): python3-numpy-f2py-1.14.3-10.el8.x86_64. 2.0 MB/s | 225 kB 00:00 2024-01-27T23:22:39.799 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-27T23:22:39.835 INFO:teuthology.orchestra.run.smithi194.stdout:(5/8): python3-numpy-1.14.3-10.el8.x86_64.rpm 4.5 MB/s | 3.7 MB 00:00 2024-01-27T23:22:40.349 INFO:teuthology.orchestra.run.smithi194.stdout:(6/8): python3-scipy-1.0.0-21.module_el8+762+77 22 MB/s | 14 MB 00:00 2024-01-27T23:22:40.374 INFO:teuthology.orchestra.run.smithi194.stdout:(7/8): libquadmath-8.5.0-21.el8.x86_64.rpm 320 kB/s | 172 kB 00:00 2024-01-27T23:22:40.396 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-27T23:22:40.615 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-27T23:22:40.691 INFO:teuthology.orchestra.run.smithi194.stdout:(8/8): libgfortran-8.5.0-21.el8.x86_64.rpm 679 kB/s | 645 kB 00:00 2024-01-27T23:22:40.691 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:40.692 INFO:teuthology.orchestra.run.smithi194.stdout:Total 17 MB/s | 37 MB 00:02 2024-01-27T23:22:40.923 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:22:40.941 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:22:40.941 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:22:41.241 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-27T23:22:41.256 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:22:41.259 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:22:41.429 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-27T23:22:42.114 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-27T23:22:42.119 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:22:42.270 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-27T23:22:42.405 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libquadmath-8.5.0-21.el8.x86_64 1/8 2024-01-27T23:22:42.468 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-27T23:22:42.545 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-27T23:22:42.748 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-27T23:22:43.123 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libgfortran-8.5.0-21.el8.x86_64 2/8 2024-01-27T23:22:43.336 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-0.3.15-4.el8.x86_64 3/8 2024-01-27T23:22:43.934 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-0.3.15-4.el8.x86_64 3/8 2024-01-27T23:22:44.122 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-27T23:22:44.504 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-27T23:22:44.804 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: openblas-threads-0.3.15-4.el8.x86_64 4/8 2024-01-27T23:22:45.166 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-1:1.14.3-10.el8.x86_64 5/8 2024-01-27T23:22:45.428 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-27T23:22:45.805 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-27T23:22:46.655 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-27T23:22:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-27T23:22:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-27T23:22:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-27T23:22:46.985 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-27T23:22:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-27T23:22:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-27T23:22:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-27T23:22:46.986 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-27T23:22:47.203 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 6/8 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-27T23:22:47.400 INFO:teuthology.orchestra.run.smithi121.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-27T23:22:47.401 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-27T23:22:47.401 INFO:teuthology.orchestra.run.smithi121.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-27T23:22:47.401 INFO:teuthology.orchestra.run.smithi121.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-27T23:22:47.401 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:47.401 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:47.520 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-rook 2024-01-27T23:22:48.277 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:45 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:49.157 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 7/8 2024-01-27T23:22:49.372 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-27T23:22:49.664 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 8/8 2024-01-27T23:22:49.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 1/8 2024-01-27T23:22:49.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-0.3.15-4.el8.x86_64 2/8 2024-01-27T23:22:49.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : openblas-threads-0.3.15-4.el8.x86_64 3/8 2024-01-27T23:22:49.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-1:1.14.3-10.el8.x86_64 4/8 2024-01-27T23:22:49.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-numpy-f2py-1:1.14.3-10.el8.x86_64 5/8 2024-01-27T23:22:49.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_6 6/8 2024-01-27T23:22:49.665 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libgfortran-8.5.0-21.el8.x86_64 7/8 2024-01-27T23:22:49.767 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:49.769 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-27T23:22:49.770 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout:Install 13 Packages 2024-01-27T23:22:49.771 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:49.772 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 1.9 M 2024-01-27T23:22:49.772 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 24 M 2024-01-27T23:22:49.772 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:22:50.082 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libquadmath-8.5.0-21.el8.x86_64 8/8 2024-01-27T23:22:50.082 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:50.082 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:22:50.082 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-diskprediction-local-2:16.2.4-0.el8.noarch 2024-01-27T23:22:50.082 INFO:teuthology.orchestra.run.smithi194.stdout: libgfortran-8.5.0-21.el8.x86_64 2024-01-27T23:22:50.082 INFO:teuthology.orchestra.run.smithi194.stdout: libquadmath-8.5.0-21.el8.x86_64 2024-01-27T23:22:50.083 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-0.3.15-4.el8.x86_64 2024-01-27T23:22:50.083 INFO:teuthology.orchestra.run.smithi194.stdout: openblas-threads-0.3.15-4.el8.x86_64 2024-01-27T23:22:50.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-1:1.14.3-10.el8.x86_64 2024-01-27T23:22:50.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-numpy-f2py-1:1.14.3-10.el8.x86_64 2024-01-27T23:22:50.083 INFO:teuthology.orchestra.run.smithi194.stdout: python3-scipy-1.0.0-21.module_el8+762+77bd8591.x86_64 2024-01-27T23:22:50.083 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:50.083 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:50.115 INFO:teuthology.orchestra.run.smithi121.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 601 kB/s | 27 kB 00:00 2024-01-27T23:22:50.192 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-rook 2024-01-27T23:22:50.265 INFO:teuthology.orchestra.run.smithi121.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 102 kB/s | 20 kB 00:00 2024-01-27T23:22:50.282 INFO:teuthology.orchestra.run.smithi121.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 125 kB/s | 27 kB 00:00 2024-01-27T23:22:50.449 INFO:teuthology.orchestra.run.smithi121.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 378 kB/s | 126 kB 00:00 2024-01-27T23:22:50.474 INFO:teuthology.orchestra.run.smithi121.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 527 kB/s | 110 kB 00:00 2024-01-27T23:22:50.633 INFO:teuthology.orchestra.run.smithi121.stdout:(6/13): python3-cachetools-3.1.1-4.el8.noarch.r 208 kB/s | 33 kB 00:00 2024-01-27T23:22:50.658 INFO:teuthology.orchestra.run.smithi121.stdout:(7/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 412 kB/s | 155 kB 00:00 2024-01-27T23:22:50.692 INFO:teuthology.orchestra.run.smithi121.stdout:(8/13): python3-certifi-2018.10.15-7.el8.noarch 278 kB/s | 16 kB 00:00 2024-01-27T23:22:50.775 INFO:teuthology.orchestra.run.smithi121.stdout:(9/13): python3-requests-oauthlib-1.0.0-1.el8.n 131 kB/s | 43 kB 00:00 2024-01-27T23:22:50.867 INFO:teuthology.orchestra.run.smithi121.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 389 kB/s | 81 kB 00:00 2024-01-27T23:22:50.926 INFO:teuthology.orchestra.run.smithi121.stdout:(11/13): python3-websocket-client-0.56.0-5.el8. 1.0 MB/s | 61 kB 00:00 2024-01-27T23:22:50.947 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:46 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:22:50.993 INFO:teuthology.orchestra.run.smithi121.stdout:(12/13): python3-kubernetes-11.0.0-6.el8.noarch 3.9 MB/s | 1.2 MB 00:00 2024-01-27T23:22:51.018 INFO:teuthology.orchestra.run.smithi121.stdout:(13/13): python3-rsa-4.9-2.el8.noarch.rpm 265 kB/s | 64 kB 00:00 2024-01-27T23:22:51.018 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:51.018 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.5 MB/s | 1.9 MB 00:01 2024-01-27T23:22:51.069 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:22:51.084 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:22:51.085 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:22:51.170 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:22:51.173 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:22:51.674 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:22:51.995 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-27T23:22:52.179 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-27T23:22:52.448 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-27T23:22:52.452 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:22:52.453 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook noarch 2:16.2.4-0.el8 ceph-noarch 27 k 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools noarch 3.1.1-4.el8 epel 33 k 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi noarch 2018.10.15-7.el8 epel 16 k 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth noarch 1:1.1.1-10.el8 epel 81 k 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch noarch 1.21-2.el8 appstream 27 k 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer noarch 1.10-11.el8 appstream 20 k 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes noarch 1:11.0.0-6.el8 epel 1.2 M 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib noarch 2.1.0-1.el8 baseos 155 k 2024-01-27T23:22:52.454 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1 noarch 0.3.7-6.el8 appstream 126 k 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules noarch 0.3.7-6.el8 appstream 110 k 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib noarch 1.0.0-1.el8 baseos 43 k 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa noarch 4.9-2.el8 epel 64 k 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client noarch 0.56.0-5.el8 epel 61 k 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout:Install 13 Packages 2024-01-27T23:22:52.455 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:52.456 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 1.9 M 2024-01-27T23:22:52.456 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 24 M 2024-01-27T23:22:52.456 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:22:52.694 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-27T23:22:52.885 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-27T23:22:52.898 INFO:teuthology.orchestra.run.smithi194.stdout:(1/13): ceph-mgr-rook-16.2.4-0.el8.noarch.rpm 632 kB/s | 27 kB 00:00 2024-01-27T23:22:53.048 INFO:teuthology.orchestra.run.smithi194.stdout:(2/13): python3-jsonpointer-1.10-11.el8.noarch. 104 kB/s | 20 kB 00:00 2024-01-27T23:22:53.073 INFO:teuthology.orchestra.run.smithi194.stdout:(3/13): python3-jsonpatch-1.21-2.el8.noarch.rpm 122 kB/s | 27 kB 00:00 2024-01-27T23:22:53.134 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-27T23:22:53.282 INFO:teuthology.orchestra.run.smithi194.stdout:(4/13): python3-pyasn1-0.3.7-6.el8.noarch.rpm 329 kB/s | 126 kB 00:00 2024-01-27T23:22:53.366 INFO:teuthology.orchestra.run.smithi194.stdout:(5/13): python3-pyasn1-modules-0.3.7-6.el8.noar 347 kB/s | 110 kB 00:00 2024-01-27T23:22:53.445 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-27T23:22:53.583 INFO:teuthology.orchestra.run.smithi194.stdout:(6/13): python3-oauthlib-2.1.0-1.el8.noarch.rpm 304 kB/s | 155 kB 00:00 2024-01-27T23:22:53.633 INFO:teuthology.orchestra.run.smithi194.stdout:(7/13): python3-requests-oauthlib-1.0.0-1.el8.n 122 kB/s | 43 kB 00:00 2024-01-27T23:22:53.692 INFO:teuthology.orchestra.run.smithi194.stdout:(8/13): python3-cachetools-3.1.1-4.el8.noarch.r 101 kB/s | 33 kB 00:00 2024-01-27T23:22:53.693 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-27T23:22:53.892 INFO:teuthology.orchestra.run.smithi194.stdout:(9/13): python3-certifi-2018.10.15-7.el8.noarch 53 kB/s | 16 kB 00:00 2024-01-27T23:22:53.993 INFO:teuthology.orchestra.run.smithi194.stdout:(10/13): python3-google-auth-1.1.1-10.el8.noarc 226 kB/s | 81 kB 00:00 2024-01-27T23:22:54.092 INFO:teuthology.orchestra.run.smithi194.stdout:(11/13): python3-rsa-4.9-2.el8.noarch.rpm 321 kB/s | 64 kB 00:00 2024-01-27T23:22:54.118 INFO:teuthology.orchestra.run.smithi194.stdout:(12/13): python3-websocket-client-0.56.0-5.el8. 490 kB/s | 61 kB 00:00 2024-01-27T23:22:54.210 INFO:teuthology.orchestra.run.smithi194.stdout:(13/13): python3-kubernetes-11.0.0-6.el8.noarch 2.3 MB/s | 1.2 MB 00:00 2024-01-27T23:22:54.210 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:54.210 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.1 MB/s | 1.9 MB 00:01 2024-01-27T23:22:54.268 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:22:54.284 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:22:54.284 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:22:54.374 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:22:54.377 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:22:54.757 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-27T23:22:54.885 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:22:55.044 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-27T23:22:55.146 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-0.3.7-6.el8.noarch 1/13 2024-01-27T23:22:55.278 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-27T23:22:55.376 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-pyasn1-modules-0.3.7-6.el8.noarch 2/13 2024-01-27T23:22:55.462 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-27T23:22:55.587 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-rsa-4.9-2.el8.noarch 3/13 2024-01-27T23:22:55.637 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-27T23:22:55.858 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-websocket-client-0.56.0-5.el8.noarch 4/13 2024-01-27T23:22:55.933 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-27T23:22:55.933 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-27T23:22:55.933 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-27T23:22:55.934 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-27T23:22:56.083 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-certifi-2018.10.15-7.el8.noarch 5/13 2024-01-27T23:22:56.341 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-cachetools-3.1.1-4.el8.noarch 6/13 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-27T23:22:56.407 INFO:teuthology.orchestra.run.smithi121.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-27T23:22:56.408 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-27T23:22:56.408 INFO:teuthology.orchestra.run.smithi121.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-27T23:22:56.408 INFO:teuthology.orchestra.run.smithi121.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-27T23:22:56.408 INFO:teuthology.orchestra.run.smithi121.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-27T23:22:56.408 INFO:teuthology.orchestra.run.smithi121.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-27T23:22:56.408 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:56.408 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:22:56.519 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-mgr-cephadm 2024-01-27T23:22:56.666 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-google-auth-1:1.1.1-10.el8.noarch 7/13 2024-01-27T23:22:56.915 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-oauthlib-2.1.0-1.el8.noarch 8/13 2024-01-27T23:22:57.263 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:01:54 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:22:57.942 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-requests-oauthlib-1.0.0-1.el8.noarch 9/13 2024-01-27T23:22:58.199 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-kubernetes-1:11.0.0-6.el8.noarch 10/13 2024-01-27T23:22:58.474 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpointer-1.10-11.el8.noarch 11/13 2024-01-27T23:22:58.683 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jsonpatch-1.21-2.el8.noarch 12/13 2024-01-27T23:22:58.770 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:22:58.771 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: Package Arch Version Repository Size 2024-01-27T23:22:58.771 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:58.771 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:22:58.771 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout:Install 6 Packages 2024-01-27T23:22:58.772 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:22:58.773 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 5.8 M 2024-01-27T23:22:58.773 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 24 M 2024-01-27T23:22:58.773 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:22:58.842 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-27T23:22:58.909 INFO:teuthology.orchestra.run.smithi121.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 2.0 MB/s | 89 kB 00:00 2024-01-27T23:22:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-rook-2:16.2.4-0.el8.noarch 13/13 2024-01-27T23:22:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-rook-2:16.2.4-0.el8.noarch 1/13 2024-01-27T23:22:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpatch-1.21-2.el8.noarch 2/13 2024-01-27T23:22:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jsonpointer-1.10-11.el8.noarch 3/13 2024-01-27T23:22:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-0.3.7-6.el8.noarch 4/13 2024-01-27T23:22:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-pyasn1-modules-0.3.7-6.el8.noarch 5/13 2024-01-27T23:22:59.139 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-oauthlib-2.1.0-1.el8.noarch 6/13 2024-01-27T23:22:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-requests-oauthlib-1.0.0-1.el8.noarch 7/13 2024-01-27T23:22:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-cachetools-3.1.1-4.el8.noarch 8/13 2024-01-27T23:22:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-certifi-2018.10.15-7.el8.noarch 9/13 2024-01-27T23:22:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-google-auth-1:1.1.1-10.el8.noarch 10/13 2024-01-27T23:22:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-kubernetes-1:11.0.0-6.el8.noarch 11/13 2024-01-27T23:22:59.140 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-rsa-4.9-2.el8.noarch 12/13 2024-01-27T23:22:59.201 INFO:teuthology.orchestra.run.smithi121.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 73 kB/s | 21 kB 00:00 2024-01-27T23:22:59.352 INFO:teuthology.orchestra.run.smithi121.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.1 MB/s | 538 kB 00:00 2024-01-27T23:22:59.594 INFO:teuthology.orchestra.run.smithi121.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.6 MB/s | 4.8 MB 00:00 2024-01-27T23:22:59.595 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-websocket-client-0.56.0-5.el8.noarch 13/13 2024-01-27T23:22:59.595 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:59.595 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:22:59.595 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-rook-2:16.2.4-0.el8.noarch 2024-01-27T23:22:59.595 INFO:teuthology.orchestra.run.smithi194.stdout: python3-cachetools-3.1.1-4.el8.noarch 2024-01-27T23:22:59.595 INFO:teuthology.orchestra.run.smithi194.stdout: python3-certifi-2018.10.15-7.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-google-auth-1:1.1.1-10.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpatch-1.21-2.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jsonpointer-1.10-11.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-kubernetes-1:11.0.0-6.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-oauthlib-2.1.0-1.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-0.3.7-6.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-pyasn1-modules-0.3.7-6.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-requests-oauthlib-1.0.0-1.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-rsa-4.9-2.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: python3-websocket-client-0.56.0-5.el8.noarch 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:22:59.596 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:22:59.619 INFO:teuthology.orchestra.run.smithi121.stdout:(5/6): python3-execnet-1.7.1-1.el8.noarch.rpm 852 kB/s | 355 kB 00:00 2024-01-27T23:22:59.677 INFO:teuthology.orchestra.run.smithi121.stdout:(6/6): python3-remoto-1.2.1-1.el8.noarch.rpm 150 kB/s | 49 kB 00:00 2024-01-27T23:22:59.678 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:22:59.678 INFO:teuthology.orchestra.run.smithi121.stdout:Total 6.4 MB/s | 5.8 MB 00:00 2024-01-27T23:22:59.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-mgr-cephadm 2024-01-27T23:22:59.728 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:22:59.737 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:22:59.738 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:22:59.835 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:22:59.839 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:23:00.130 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:23:00.409 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-27T23:23:00.461 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:01:56 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:00.713 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-27T23:23:01.624 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-27T23:23:01.967 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:01.968 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:01.968 INFO:teuthology.orchestra.run.smithi194.stdout: Package Arch Version Repository Size 2024-01-27T23:23:01.968 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:01.968 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:23:01.968 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm noarch 2:16.2.4-0.el8 ceph-noarch 89 k 2024-01-27T23:23:01.968 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:23:01.968 INFO:teuthology.orchestra.run.smithi194.stdout: python3-apipkg noarch 1.5-6.el8 epel 21 k 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel noarch 2.5.1-7.el8 appstream 4.8 M 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout: python3-execnet noarch 1.7.1-1.el8 epel 355 k 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2 noarch 2.10.1-3.el8 appstream 538 k 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout: python3-remoto noarch 1.2.1-1.el8 epel 49 k 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout:Install 6 Packages 2024-01-27T23:23:01.969 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:01.970 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 5.8 M 2024-01-27T23:23:01.970 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 24 M 2024-01-27T23:23:01.970 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:23:01.980 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-27T23:23:02.374 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-27T23:23:02.594 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:23:02.649 INFO:teuthology.orchestra.run.smithi194.stdout:(1/6): ceph-mgr-cephadm-16.2.4-0.el8.noarch.rpm 1.8 MB/s | 89 kB 00:00 2024-01-27T23:23:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:23:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-27T23:23:02.888 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-27T23:23:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-27T23:23:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-27T23:23:02.889 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-27T23:23:02.975 INFO:teuthology.orchestra.run.smithi194.stdout:(2/6): python3-apipkg-1.5-6.el8.noarch.rpm 65 kB/s | 21 kB 00:00 2024-01-27T23:23:03.050 INFO:teuthology.orchestra.run.smithi194.stdout:(3/6): python3-jinja2-2.10.1-3.el8.noarch.rpm 1.2 MB/s | 538 kB 00:00 2024-01-27T23:23:03.280 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-27T23:23:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:03.281 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:23:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-27T23:23:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-27T23:23:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-27T23:23:03.281 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:03.281 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:03.317 INFO:teuthology.orchestra.run.smithi194.stdout:(4/6): python3-babel-2.5.1-7.el8.noarch.rpm 6.7 MB/s | 4.8 MB 00:00 2024-01-27T23:23:03.413 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install ceph-fuse 2024-01-27T23:23:03.434 INFO:teuthology.orchestra.run.smithi194.stdout:(5/6): python3-remoto-1.2.1-1.el8.noarch.rpm 127 kB/s | 49 kB 00:00 2024-01-27T23:23:03.459 INFO:teuthology.orchestra.run.smithi194.stdout:(6/6): python3-execnet-1.7.1-1.el8.noarch.rpm 734 kB/s | 355 kB 00:00 2024-01-27T23:23:03.464 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:03.464 INFO:teuthology.orchestra.run.smithi194.stdout:Total 3.9 MB/s | 5.8 MB 00:01 2024-01-27T23:23:03.540 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:23:03.550 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:23:03.551 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:23:03.662 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:23:03.665 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:23:04.127 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:23:04.186 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:01 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:04.407 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-apipkg-1.5-6.el8.noarch 1/6 2024-01-27T23:23:04.710 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-execnet-1.7.1-1.el8.noarch 2/6 2024-01-27T23:23:05.658 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-remoto-1.2.1-1.el8.noarch 3/6 2024-01-27T23:23:05.687 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout:Installing dependencies: 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout:Install 2 Packages 2024-01-27T23:23:05.688 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:05.689 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 968 k 2024-01-27T23:23:05.689 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 3.2 M 2024-01-27T23:23:05.689 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:23:06.036 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-babel-2.5.1-7.el8.noarch 4/6 2024-01-27T23:23:06.287 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : python3-jinja2-2.10.1-3.el8.noarch 5/6 2024-01-27T23:23:06.466 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:23:06.750 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 6/6 2024-01-27T23:23:06.751 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-mgr-cephadm-2:16.2.4-0.el8.noarch 1/6 2024-01-27T23:23:06.751 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-babel-2.5.1-7.el8.noarch 2/6 2024-01-27T23:23:06.751 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-jinja2-2.10.1-3.el8.noarch 3/6 2024-01-27T23:23:06.751 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-apipkg-1.5-6.el8.noarch 4/6 2024-01-27T23:23:06.751 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-execnet-1.7.1-1.el8.noarch 5/6 2024-01-27T23:23:06.987 INFO:teuthology.orchestra.run.smithi121.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2024-01-27T23:23:07.134 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : python3-remoto-1.2.1-1.el8.noarch 6/6 2024-01-27T23:23:07.134 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:07.134 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:23:07.134 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-mgr-cephadm-2:16.2.4-0.el8.noarch python3-apipkg-1.5-6.el8.noarch 2024-01-27T23:23:07.135 INFO:teuthology.orchestra.run.smithi194.stdout: python3-babel-2.5.1-7.el8.noarch python3-execnet-1.7.1-1.el8.noarch 2024-01-27T23:23:07.135 INFO:teuthology.orchestra.run.smithi194.stdout: python3-jinja2-2.10.1-3.el8.noarch python3-remoto-1.2.1-1.el8.noarch 2024-01-27T23:23:07.135 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:07.135 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:07.263 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install ceph-fuse 2024-01-27T23:23:07.312 INFO:teuthology.orchestra.run.smithi121.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 213 kB/s | 83 kB 00:00 2024-01-27T23:23:07.313 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:07.314 INFO:teuthology.orchestra.run.smithi121.stdout:Total 596 kB/s | 968 kB 00:01 2024-01-27T23:23:07.325 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:23:07.347 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:23:07.347 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:23:07.442 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:23:07.444 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:23:07.666 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:23:08.020 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:04 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:08.331 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-27T23:23:08.526 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-27T23:23:08.932 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-27T23:23:08.932 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-27T23:23:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-27T23:23:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:09.286 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:23:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-27T23:23:09.286 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:09.287 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:09.522 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados-devel 2024-01-27T23:23:09.532 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:09.532 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse x86_64 2:16.2.4-0.el8 ceph 885 k 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout:Installing dependencies: 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout: fuse x86_64 2.9.7-17.el8 baseos 83 k 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout:Install 2 Packages 2024-01-27T23:23:09.533 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:09.534 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 968 k 2024-01-27T23:23:09.534 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 3.2 M 2024-01-27T23:23:09.534 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:23:10.285 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:07 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:10.803 INFO:teuthology.orchestra.run.smithi194.stdout:(1/2): ceph-fuse-16.2.4-0.el8.x86_64.rpm 13 MB/s | 885 kB 00:00 2024-01-27T23:23:11.070 INFO:teuthology.orchestra.run.smithi194.stdout:(2/2): fuse-2.9.7-17.el8.x86_64.rpm 250 kB/s | 83 kB 00:00 2024-01-27T23:23:11.070 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:11.071 INFO:teuthology.orchestra.run.smithi194.stdout:Total 630 kB/s | 968 kB 00:01 2024-01-27T23:23:11.082 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:23:11.104 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:23:11.105 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:23:11.196 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:23:11.199 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:23:11.390 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:23:11.698 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : fuse-2.9.7-17.el8.x86_64 1/2 2024-01-27T23:23:11.784 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:11.787 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 131 k 2024-01-27T23:23:11.788 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 439 k 2024-01-27T23:23:11.789 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:23:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 3.2 MB/s | 131 kB 00:00 2024-01-27T23:23:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.1 MB/s | 131 kB 00:00 2024-01-27T23:23:11.827 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:23:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:23:11.833 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:23:11.882 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:23:11.885 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:23:11.898 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-27T23:23:12.019 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:23:12.233 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:12.326 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: ceph-fuse-2:16.2.4-0.el8.x86_64 2/2 2024-01-27T23:23:12.326 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : ceph-fuse-2:16.2.4-0.el8.x86_64 1/2 2024-01-27T23:23:12.539 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:12.744 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : fuse-2.9.7-17.el8.x86_64 2/2 2024-01-27T23:23:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:12.745 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:23:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: ceph-fuse-2:16.2.4-0.el8.x86_64 fuse-2.9.7-17.el8.x86_64 2024-01-27T23:23:12.745 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:12.745 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:12.936 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:12.937 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:12.937 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:23:12.937 INFO:teuthology.orchestra.run.smithi121.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:12.937 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:12.937 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:12.969 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados-devel 2024-01-27T23:23:13.158 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs2 2024-01-27T23:23:13.736 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:09 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:13.919 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:10 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:15.234 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel x86_64 2:16.2.4-0.el8 ceph 131 k 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:15.235 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-27T23:23:15.236 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:15.236 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 131 k 2024-01-27T23:23:15.236 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 439 k 2024-01-27T23:23:15.236 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:23:15.291 INFO:teuthology.orchestra.run.smithi194.stdout:librados-devel-16.2.4-0.el8.x86_64.rpm 2.4 MB/s | 131 kB 00:00 2024-01-27T23:23:15.291 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:15.291 INFO:teuthology.orchestra.run.smithi194.stdout:Total 2.3 MB/s | 131 kB 00:00 2024-01-27T23:23:15.292 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:23:15.297 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:23:15.297 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:23:15.347 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:23:15.350 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:23:15.365 INFO:teuthology.orchestra.run.smithi121.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:15.411 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:15.413 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:23:15.413 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:15.463 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:23:15.667 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install libcephfs-devel 2024-01-27T23:23:15.705 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:16.026 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:16.394 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : librados-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:16.394 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:16.394 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:23:16.394 INFO:teuthology.orchestra.run.smithi194.stdout: librados-devel-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:16.394 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:16.394 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:16.428 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:13 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:16.613 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs2 2024-01-27T23:23:17.384 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:13 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:17.927 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:17.927 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:17.927 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:17.927 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:17.927 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:23:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-27T23:23:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:17.928 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:23:17.928 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:17.928 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-01-27T23:23:17.928 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:17.928 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 24 k 2024-01-27T23:23:17.929 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 94 k 2024-01-27T23:23:17.929 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:23:17.959 INFO:teuthology.orchestra.run.smithi121.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 801 kB/s | 24 kB 00:00 2024-01-27T23:23:17.959 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:17.960 INFO:teuthology.orchestra.run.smithi121.stdout:Total 769 kB/s | 24 kB 00:00 2024-01-27T23:23:17.960 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:23:17.963 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:23:17.963 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:23:17.966 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:23:17.969 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:23:18.019 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:23:18.231 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:18.492 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:18.834 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:18.835 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:23:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:18.835 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:18.835 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:18.835 INFO:teuthology.orchestra.run.smithi194.stdout:Package libcephfs2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:18.881 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:18.882 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:23:18.882 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:18.954 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librados2 2024-01-27T23:23:18.970 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install libcephfs-devel 2024-01-27T23:23:19.707 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:16 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:19.721 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:15 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:21.176 INFO:teuthology.orchestra.run.smithi121.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:21.218 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel x86_64 2:16.2.4-0.el8 ceph 24 k 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-27T23:23:21.219 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:21.220 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 24 k 2024-01-27T23:23:21.220 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 94 k 2024-01-27T23:23:21.220 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:23:21.221 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:21.223 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:23:21.223 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:21.279 INFO:teuthology.orchestra.run.smithi194.stdout:libcephfs-devel-16.2.4-0.el8.x86_64.rpm 413 kB/s | 24 kB 00:00 2024-01-27T23:23:21.279 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:21.280 INFO:teuthology.orchestra.run.smithi194.stdout:Total 404 kB/s | 24 kB 00:00 2024-01-27T23:23:21.280 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:23:21.283 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:23:21.283 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:23:21.286 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:23:21.289 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:23:21.321 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install librbd1 2024-01-27T23:23:21.338 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:23:21.512 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:21.803 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:22.072 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:19 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:22.144 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : libcephfs-devel-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:22.145 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:22.145 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:23:22.145 INFO:teuthology.orchestra.run.smithi194.stdout: libcephfs-devel-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:22.145 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:22.145 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:22.252 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librados2 2024-01-27T23:23:23.004 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:19 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:23.530 INFO:teuthology.orchestra.run.smithi121.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:23.576 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:23.577 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:23:23.577 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:23.695 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rados 2024-01-27T23:23:24.451 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:21 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:24.454 INFO:teuthology.orchestra.run.smithi194.stdout:Package librados2-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:24.501 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:24.502 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:23:24.502 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:24.601 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install librbd1 2024-01-27T23:23:25.357 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:21 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:25.910 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:25.955 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:25.956 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:23:25.957 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:26.068 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rgw 2024-01-27T23:23:26.790 INFO:teuthology.orchestra.run.smithi194.stdout:Package librbd1-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:26.817 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:23 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:26.835 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:26.836 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:23:26.836 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:26.929 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rados 2024-01-27T23:23:27.683 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:23 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:28.276 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:28.322 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:23:28.323 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:28.427 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-cephfs 2024-01-27T23:23:29.150 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rados-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:29.173 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:26 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:29.196 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:29.196 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:23:29.197 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:29.298 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rgw 2024-01-27T23:23:30.051 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:26 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:30.631 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:30.676 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:30.677 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:23:30.677 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:30.776 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install python3-rbd 2024-01-27T23:23:31.502 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rgw-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:31.525 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:28 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:31.547 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:31.548 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:23:31.549 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:31.651 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-cephfs 2024-01-27T23:23:32.396 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:28 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:32.981 INFO:teuthology.orchestra.run.smithi121.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:33.027 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:33.028 INFO:teuthology.orchestra.run.smithi121.stdout:Nothing to do. 2024-01-27T23:23:33.028 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:33.124 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-fuse 2024-01-27T23:23:33.851 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-cephfs-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:33.881 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:30 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:33.896 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:33.897 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:23:33.898 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:33.997 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install python3-rbd 2024-01-27T23:23:34.752 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:30 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:35.437 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 89 k 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 298 k 2024-01-27T23:23:35.438 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:23:35.489 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.7 MB/s | 89 kB 00:00 2024-01-27T23:23:35.490 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:35.490 INFO:teuthology.orchestra.run.smithi121.stdout:Total 1.7 MB/s | 89 kB 00:00 2024-01-27T23:23:35.490 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:23:35.500 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:23:35.500 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:23:35.550 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:23:35.553 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:23:35.669 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:23:35.861 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:36.153 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:36.207 INFO:teuthology.orchestra.run.smithi194.stdout:Package python3-rbd-2:16.2.4-0.el8.x86_64 is already installed. 2024-01-27T23:23:36.253 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:36.254 INFO:teuthology.orchestra.run.smithi194.stdout:Nothing to do. 2024-01-27T23:23:36.254 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:36.345 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-fuse 2024-01-27T23:23:36.565 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:36.565 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:36.565 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:23:36.565 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:36.565 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:36.565 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:36.741 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-mirror 2024-01-27T23:23:37.102 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:33 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:37.501 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:34 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:38.607 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:38.609 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:38.609 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:38.609 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:38.609 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:23:38.610 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse x86_64 2:16.2.4-0.el8 ceph 89 k 2024-01-27T23:23:38.610 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:38.610 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:23:38.610 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:38.610 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-27T23:23:38.610 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:38.611 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 89 k 2024-01-27T23:23:38.611 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 298 k 2024-01-27T23:23:38.611 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:23:38.693 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-fuse-16.2.4-0.el8.x86_64.rpm 1.0 MB/s | 89 kB 00:00 2024-01-27T23:23:38.693 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:38.693 INFO:teuthology.orchestra.run.smithi194.stdout:Total 1.0 MB/s | 89 kB 00:00 2024-01-27T23:23:38.694 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:23:38.703 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:23:38.703 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:23:38.753 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:23:38.756 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:23:38.864 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:23:38.968 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:38.968 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:38.968 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:38.968 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:38.969 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 3.7 M 2024-01-27T23:23:38.970 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 18 M 2024-01-27T23:23:38.970 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:23:39.059 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 42 MB/s | 3.7 MB 00:00 2024-01-27T23:23:39.059 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:39.060 INFO:teuthology.orchestra.run.smithi121.stdout:Total 41 MB/s | 3.7 MB 00:00 2024-01-27T23:23:39.060 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:23:39.067 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:39.072 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:23:39.072 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:23:39.134 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:23:39.137 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:23:39.390 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:39.601 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:23:39.779 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-fuse-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:39.779 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:39.779 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:23:39.779 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-fuse-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:39.779 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:39.779 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:39.803 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:40.032 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-mirror 2024-01-27T23:23:40.225 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:23:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:40.580 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:40.580 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:40.795 DEBUG:teuthology.orchestra.run.smithi121:> sudo yum -y install rbd-nbd 2024-01-27T23:23:40.798 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:36 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:41.574 INFO:teuthology.orchestra.run.smithi121.stdout:Last metadata expiration check: 0:02:38 ago on Sat 27 Jan 2024 11:21:03 PM UTC. 2024-01-27T23:23:42.291 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror x86_64 2:16.2.4-0.el8 ceph 3.7 M 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-27T23:23:42.292 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:42.293 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 3.7 M 2024-01-27T23:23:42.293 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 18 M 2024-01-27T23:23:42.293 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:23:42.378 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-mirror-16.2.4-0.el8.x86_64.rpm 44 MB/s | 3.7 MB 00:00 2024-01-27T23:23:42.379 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:42.379 INFO:teuthology.orchestra.run.smithi194.stdout:Total 43 MB/s | 3.7 MB 00:00 2024-01-27T23:23:42.379 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:23:42.391 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:23:42.391 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:23:42.453 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:23:42.456 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:23:42.920 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:23:43.092 INFO:teuthology.orchestra.run.smithi121.stdout:Dependencies resolved. 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout:Installing: 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction Summary 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout:================================================================================ 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout:Install 1 Package 2024-01-27T23:23:43.093 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:43.094 INFO:teuthology.orchestra.run.smithi121.stdout:Total download size: 173 k 2024-01-27T23:23:43.094 INFO:teuthology.orchestra.run.smithi121.stdout:Installed size: 649 k 2024-01-27T23:23:43.094 INFO:teuthology.orchestra.run.smithi121.stdout:Downloading Packages: 2024-01-27T23:23:43.127 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:43.137 INFO:teuthology.orchestra.run.smithi121.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 3.9 MB/s | 173 kB 00:00 2024-01-27T23:23:43.138 INFO:teuthology.orchestra.run.smithi121.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:43.138 INFO:teuthology.orchestra.run.smithi121.stdout:Total 3.8 MB/s | 173 kB 00:00 2024-01-27T23:23:43.138 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction check 2024-01-27T23:23:43.149 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction check succeeded. 2024-01-27T23:23:43.149 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction test 2024-01-27T23:23:43.200 INFO:teuthology.orchestra.run.smithi121.stdout:Transaction test succeeded. 2024-01-27T23:23:43.203 INFO:teuthology.orchestra.run.smithi121.stdout:Running transaction 2024-01-27T23:23:43.320 INFO:teuthology.orchestra.run.smithi121.stdout: Preparing : 1/1 2024-01-27T23:23:43.542 INFO:teuthology.orchestra.run.smithi121.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:43.552 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:43.835 INFO:teuthology.orchestra.run.smithi121.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:43.919 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-mirror-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:43.919 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:43.919 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:23:43.920 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-mirror-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:43.920 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:43.920 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:44.116 DEBUG:teuthology.orchestra.run.smithi194:> sudo yum -y install rbd-nbd 2024-01-27T23:23:44.240 INFO:teuthology.orchestra.run.smithi121.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:44.240 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:44.241 INFO:teuthology.orchestra.run.smithi121.stdout:Installed: 2024-01-27T23:23:44.241 INFO:teuthology.orchestra.run.smithi121.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:44.241 INFO:teuthology.orchestra.run.smithi121.stdout: 2024-01-27T23:23:44.241 INFO:teuthology.orchestra.run.smithi121.stdout:Complete! 2024-01-27T23:23:44.446 DEBUG:teuthology.parallel:result is None 2024-01-27T23:23:44.887 INFO:teuthology.orchestra.run.smithi194.stdout:Last metadata expiration check: 0:02:40 ago on Sat 27 Jan 2024 11:21:04 PM UTC. 2024-01-27T23:23:46.391 INFO:teuthology.orchestra.run.smithi194.stdout:Dependencies resolved. 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout: Package Architecture Version Repository Size 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:Installing: 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd x86_64 2:16.2.4-0.el8 ceph 173 k 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction Summary 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:================================================================================ 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:Install 1 Package 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:Total download size: 173 k 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:Installed size: 649 k 2024-01-27T23:23:46.392 INFO:teuthology.orchestra.run.smithi194.stdout:Downloading Packages: 2024-01-27T23:23:46.431 INFO:teuthology.orchestra.run.smithi194.stdout:rbd-nbd-16.2.4-0.el8.x86_64.rpm 4.4 MB/s | 173 kB 00:00 2024-01-27T23:23:46.431 INFO:teuthology.orchestra.run.smithi194.stdout:-------------------------------------------------------------------------------- 2024-01-27T23:23:46.432 INFO:teuthology.orchestra.run.smithi194.stdout:Total 4.3 MB/s | 173 kB 00:00 2024-01-27T23:23:46.432 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction check 2024-01-27T23:23:46.442 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction check succeeded. 2024-01-27T23:23:46.442 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction test 2024-01-27T23:23:46.493 INFO:teuthology.orchestra.run.smithi194.stdout:Transaction test succeeded. 2024-01-27T23:23:46.496 INFO:teuthology.orchestra.run.smithi194.stdout:Running transaction 2024-01-27T23:23:46.614 INFO:teuthology.orchestra.run.smithi194.stdout: Preparing : 1/1 2024-01-27T23:23:46.842 INFO:teuthology.orchestra.run.smithi194.stdout: Installing : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:47.143 INFO:teuthology.orchestra.run.smithi194.stdout: Running scriptlet: rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:47.543 INFO:teuthology.orchestra.run.smithi194.stdout: Verifying : rbd-nbd-2:16.2.4-0.el8.x86_64 1/1 2024-01-27T23:23:47.543 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:47.543 INFO:teuthology.orchestra.run.smithi194.stdout:Installed: 2024-01-27T23:23:47.544 INFO:teuthology.orchestra.run.smithi194.stdout: rbd-nbd-2:16.2.4-0.el8.x86_64 2024-01-27T23:23:47.544 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:23:47.544 INFO:teuthology.orchestra.run.smithi194.stdout:Complete! 2024-01-27T23:23:47.744 DEBUG:teuthology.parallel:result is None 2024-01-27T23:23:47.744 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-27T23:23:47.745 INFO:teuthology.packaging:ref: None 2024-01-27T23:23:47.745 INFO:teuthology.packaging:tag: v16.2.4 2024-01-27T23:23:47.745 INFO:teuthology.packaging:branch: None 2024-01-27T23:23:47.745 INFO:teuthology.packaging:sha1: a72a5dace2353a54147d37bd2699a2a482507966 2024-01-27T23:23:47.745 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-27T23:23:48.085 DEBUG:teuthology.orchestra.run.smithi121:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-27T23:23:48.112 INFO:teuthology.orchestra.run.smithi121.stdout:16.2.4-0.el8 2024-01-27T23:23:48.112 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-27T23:23:48.112 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-27T23:23:48.114 WARNING:teuthology.packaging:More than one of ref, tag, branch, or sha1 supplied; using tag 2024-01-27T23:23:48.114 INFO:teuthology.packaging:ref: None 2024-01-27T23:23:48.114 INFO:teuthology.packaging:tag: v16.2.4 2024-01-27T23:23:48.114 INFO:teuthology.packaging:branch: None 2024-01-27T23:23:48.114 INFO:teuthology.packaging:sha1: a72a5dace2353a54147d37bd2699a2a482507966 2024-01-27T23:23:48.115 DEBUG:teuthology.packaging:Querying https://shaman.ceph.com/api/search?status=ready&project=ceph&flavor=default&distros=centos%2F8%2Fx86_64&sha1=3cbe25cde3cfa028984618ad32de9edc4c1eaed0 2024-01-27T23:23:48.361 DEBUG:teuthology.orchestra.run.smithi194:> rpm -q ceph --qf '%{VERSION}-%{RELEASE}' 2024-01-27T23:23:48.384 INFO:teuthology.orchestra.run.smithi194.stdout:16.2.4-0.el8 2024-01-27T23:23:48.384 INFO:teuthology.packaging:The installed version of ceph is 16.2.4-0.el8 2024-01-27T23:23:48.384 INFO:teuthology.task.install:The correct ceph version 16.2.4-0 is installed. 2024-01-27T23:23:48.386 INFO:teuthology.task.install.util:Shipping valgrind.supp... 2024-01-27T23:23:48.386 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:23:48.386 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-27T23:23:48.419 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:23:48.419 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/home/ubuntu/cephtest/valgrind.supp 2024-01-27T23:23:48.452 INFO:teuthology.task.install.util:Shipping 'daemon-helper'... 2024-01-27T23:23:48.452 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:23:48.452 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/daemon-helper 2024-01-27T23:23:48.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-27T23:23:48.560 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:23:48.560 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/daemon-helper 2024-01-27T23:23:48.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/daemon-helper 2024-01-27T23:23:48.653 INFO:teuthology.task.install.util:Shipping 'adjust-ulimits'... 2024-01-27T23:23:48.654 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:23:48.654 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-27T23:23:48.683 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-27T23:23:48.751 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:23:48.752 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/adjust-ulimits 2024-01-27T23:23:48.778 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/adjust-ulimits 2024-01-27T23:23:48.847 INFO:teuthology.task.install.util:Shipping 'stdin-killer'... 2024-01-27T23:23:48.847 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:23:48.847 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd of=/usr/bin/stdin-killer 2024-01-27T23:23:48.877 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-27T23:23:48.946 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:23:48.946 DEBUG:teuthology.orchestra.run.smithi194:> sudo dd of=/usr/bin/stdin-killer 2024-01-27T23:23:48.972 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod a=rx -- /usr/bin/stdin-killer 2024-01-27T23:23:49.039 INFO:teuthology.run_tasks:Running task print... 2024-01-27T23:23:49.049 INFO:teuthology.task.print:**** done install task... 2024-01-27T23:23:49.049 INFO:teuthology.run_tasks:Running task cephadm... 2024-01-27T23:23:49.146 INFO:tasks.cephadm:Config: {'avoid_pacific_features': True, 'cephadm_branch': 'v16.2.4', 'cephadm_git_url': 'https://github.com/ceph/ceph', 'image': 'docker.io/ceph/ceph:v16.2.4', 'roleless': True, 'conf': {'client': {'client mount timeout': 600, 'debug client': 20, 'debug ms': 1, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'global': {'mon pg warn min per osd': 0}, 'mds': {'debug mds': 20, 'debug mds balancer': 20, 'debug ms': 1, 'mds debug frag': True, 'mds debug scatterstat': True, 'mds op complaint time': 180, 'mds verify scatter': True, 'osd op complaint time': 180, 'rados mon op timeout': 900, 'rados osd op timeout': 900}, 'mgr': {'debug mgr': 20, 'debug ms': 1}, 'mon': {'debug mon': 20, 'debug ms': 1, 'debug paxos': 20, 'mon op complaint time': 120}, 'osd': {'bdev async discard': True, 'bdev enable discard': True, 'bluestore allocator': 'bitmap', 'bluestore block size': 96636764160, 'bluestore fsck on mount': True, 'debug bluefs': '1/20', 'debug bluestore': '1/20', 'debug ms': 1, 'debug osd': 20, 'debug rocksdb': '4/10', 'mon osd backfillfull_ratio': 0.85, 'mon osd full ratio': 0.9, 'mon osd nearfull ratio': 0.8, 'osd failsafe full ratio': 0.95, 'osd objectstore': 'bluestore', 'osd op complaint time': 180}}, 'flavor': 'default', 'fs': 'xfs', 'log-ignorelist': ['\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', 'overall HEALTH_', '\\(FS_DEGRADED\\)', '\\(MDS_FAILED\\)', '\\(MDS_DEGRADED\\)', '\\(FS_WITH_FAILED_MDS\\)', '\\(MDS_DAMAGE\\)', '\\(MDS_ALL_DOWN\\)', '\\(MDS_UP_LESS_THAN_MAX\\)', '\\(FS_INLINE_DATA_DEPRECATED\\)', 'overall HEALTH_', '\\(OSD_DOWN\\)', '\\(OSD_', 'but it is still running', 'is not responding'], 'sha1': 'a72a5dace2353a54147d37bd2699a2a482507966'} 2024-01-27T23:23:49.146 INFO:tasks.cephadm:Cluster image is docker.io/ceph/ceph:v16.2.4 2024-01-27T23:23:49.147 INFO:tasks.cephadm:Cluster fsid is 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:23:49.147 INFO:tasks.cephadm:Choosing monitor IPs and ports... 2024-01-27T23:23:49.147 INFO:tasks.cephadm:No mon roles; fabricating mons 2024-01-27T23:23:49.147 INFO:tasks.cephadm:Monitor IPs: {'mon.smithi121': '172.21.15.121', 'mon.smithi194': '172.21.15.194'} 2024-01-27T23:23:49.147 INFO:tasks.cephadm:Normalizing hostnames... 2024-01-27T23:23:49.147 DEBUG:teuthology.orchestra.run.smithi121:> sudo hostname $(hostname -s) 2024-01-27T23:23:49.179 DEBUG:teuthology.orchestra.run.smithi194:> sudo hostname $(hostname -s) 2024-01-27T23:23:49.205 INFO:tasks.cephadm:Downloading cephadm (repo https://github.com/ceph/ceph ref v16.2.4)... 2024-01-27T23:23:49.206 DEBUG:teuthology.orchestra.run.smithi121:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-27T23:23:49.364 INFO:teuthology.orchestra.run.smithi121.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 27 23:23 /home/ubuntu/cephtest/cephadm 2024-01-27T23:23:49.364 DEBUG:teuthology.orchestra.run.smithi194:> curl --silent https://raw.githubusercontent.com/ceph/ceph/v16.2.4/src/cephadm/cephadm > /home/ubuntu/cephtest/cephadm && ls -l /home/ubuntu/cephtest/cephadm 2024-01-27T23:23:49.512 INFO:teuthology.orchestra.run.smithi194.stdout:-rw-rw-r--. 1 ubuntu ubuntu 292005 Jan 27 23:23 /home/ubuntu/cephtest/cephadm 2024-01-27T23:23:49.512 DEBUG:teuthology.orchestra.run.smithi121:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-27T23:23:49.536 DEBUG:teuthology.orchestra.run.smithi194:> test -s /home/ubuntu/cephtest/cephadm && test $(stat -c%s /home/ubuntu/cephtest/cephadm) -gt 1000 && chmod +x /home/ubuntu/cephtest/cephadm 2024-01-27T23:23:49.582 DEBUG:teuthology.orchestra.run.smithi121:> sudo mkdir -p /etc/ceph 2024-01-27T23:23:49.615 DEBUG:teuthology.orchestra.run.smithi194:> sudo mkdir -p /etc/ceph 2024-01-27T23:23:49.642 DEBUG:teuthology.orchestra.run.smithi121:> sudo chmod 777 /etc/ceph 2024-01-27T23:23:49.687 DEBUG:teuthology.orchestra.run.smithi194:> sudo chmod 777 /etc/ceph 2024-01-27T23:23:49.714 INFO:tasks.cephadm:Writing seed config... 2024-01-27T23:23:49.715 INFO:tasks.cephadm: override: [client] client mount timeout = 600 2024-01-27T23:23:49.715 INFO:tasks.cephadm: override: [client] debug client = 20 2024-01-27T23:23:49.715 INFO:tasks.cephadm: override: [client] debug ms = 1 2024-01-27T23:23:49.715 INFO:tasks.cephadm: override: [client] rados mon op timeout = 900 2024-01-27T23:23:49.715 INFO:tasks.cephadm: override: [client] rados osd op timeout = 900 2024-01-27T23:23:49.715 INFO:tasks.cephadm: override: [global] mon pg warn min per osd = 0 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] debug mds = 20 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] debug mds balancer = 20 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] debug ms = 1 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] mds debug frag = True 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] mds debug scatterstat = True 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] mds op complaint time = 180 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] mds verify scatter = True 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] osd op complaint time = 180 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] rados mon op timeout = 900 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mds] rados osd op timeout = 900 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mgr] debug mgr = 20 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mgr] debug ms = 1 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mon] debug mon = 20 2024-01-27T23:23:49.716 INFO:tasks.cephadm: override: [mon] debug ms = 1 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [mon] debug paxos = 20 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [mon] mon op complaint time = 120 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] bdev async discard = True 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] bdev enable discard = True 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] bluestore allocator = bitmap 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] bluestore block size = 96636764160 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] bluestore fsck on mount = True 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] debug bluefs = 1/20 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] debug bluestore = 1/20 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] debug ms = 1 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] debug osd = 20 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] debug rocksdb = 4/10 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] mon osd backfillfull_ratio = 0.85 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] mon osd full ratio = 0.9 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] mon osd nearfull ratio = 0.8 2024-01-27T23:23:49.717 INFO:tasks.cephadm: override: [osd] osd failsafe full ratio = 0.95 2024-01-27T23:23:49.718 INFO:tasks.cephadm: override: [osd] osd objectstore = bluestore 2024-01-27T23:23:49.718 INFO:tasks.cephadm: override: [osd] osd op complaint time = 180 2024-01-27T23:23:49.718 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:23:49.718 DEBUG:teuthology.orchestra.run.smithi121:> dd of=/home/ubuntu/cephtest/seed.ceph.conf 2024-01-27T23:23:49.749 DEBUG:tasks.cephadm:Final config: [global] # make logging friendly to teuthology log_to_file = true log_to_stderr = false mon cluster log file level = debug mon clock drift allowed = 1.000 # replicate across OSDs, not hosts osd crush chooseleaf type = 0 #osd pool default size = 2 osd pool default erasure code profile = plugin=jerasure technique=reed_sol_van k=2 m=1 ruleset-failure-domain=osd crush-failure-domain=osd # enable some debugging auth debug = true ms die on old message = true ms die on bug = true debug asserts on shutdown = true # adjust warnings mon max pg per osd = 10000# >= luminous mon pg warn max object skew = 0 mon osd allow primary affinity = true mon osd allow pg remap = true mon warn on legacy crush tunables = false mon warn on crush straw calc version zero = false mon warn on no sortbitwise = false mon warn on osd down out interval zero = false mon warn on too few osds = false mon_warn_on_pool_pg_num_not_power_of_two = false # disable pg_autoscaler by default for new pools osd_pool_default_pg_autoscale_mode = off # tests delete pools mon allow pool delete = true fsid = 2005e414-bd6b-11ee-95b2-87774f69a715 mon pg warn min per osd = 0 [osd] osd scrub load threshold = 5.0 osd scrub max interval = 600 osd recover clone overlap = true osd recovery max chunk = 1048576 osd deep scrub update digest min age = 30 osd map max advance = 10 osd memory target autotune = true # debugging osd debug shutdown = true osd debug op order = true osd debug verify stray on activate = true osd debug pg log writeout = true osd debug verify cached snaps = true osd debug verify missing on start = true osd debug misdirected ops = true osd op queue = debug_random osd op queue cut off = debug_random osd shutdown pgref assert = true bdev debug aio = true osd sloppy crc = true bdev async discard = True bdev enable discard = True bluestore allocator = bitmap bluestore block size = 96636764160 bluestore fsck on mount = True debug bluefs = 1/20 debug bluestore = 1/20 debug ms = 1 debug osd = 20 debug rocksdb = 4/10 mon osd backfillfull_ratio = 0.85 mon osd full ratio = 0.9 mon osd nearfull ratio = 0.8 osd failsafe full ratio = 0.95 osd objectstore = bluestore osd op complaint time = 180 [mgr] mon reweight min pgs per osd = 4 mon reweight min bytes per osd = 10 mgr/telemetry/nag = false debug mgr = 20 debug ms = 1 [mon] mon data avail warn = 5 mon mgr mkfs grace = 240 mon reweight min pgs per osd = 4 mon osd reporter subtree level = osd mon osd prime pg temp = true mon reweight min bytes per osd = 10 # rotate auth tickets quickly to exercise renewal paths auth mon ticket ttl = 660# 11m auth service ticket ttl = 240# 4m # don't complain about global id reclaim mon_warn_on_insecure_global_id_reclaim = false mon_warn_on_insecure_global_id_reclaim_allowed = false debug mon = 20 debug ms = 1 debug paxos = 20 mon op complaint time = 120 [client.rgw] rgw cache enabled = true rgw enable ops log = true rgw enable usage log = true [client] client mount timeout = 600 debug client = 20 debug ms = 1 rados mon op timeout = 900 rados osd op timeout = 900 [mds] debug mds = 20 debug mds balancer = 20 debug ms = 1 mds debug frag = True mds debug scatterstat = True mds op complaint time = 180 mds verify scatter = True osd op complaint time = 180 rados mon op timeout = 900 rados osd op timeout = 900 2024-01-27T23:23:49.750 DEBUG:teuthology.orchestra.run.smithi121:mon.smithi121> sudo journalctl -f -n 0 -u ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121.service 2024-01-27T23:23:49.791 INFO:tasks.cephadm:Bootstrapping... 2024-01-27T23:23:49.791 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 -v bootstrap --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 --config /home/ubuntu/cephtest/seed.ceph.conf --output-config /etc/ceph/ceph.conf --output-keyring /etc/ceph/ceph.client.admin.keyring --output-pub-ssh-key /home/ubuntu/cephtest/ceph.pub --mon-ip 172.21.15.121 && sudo chmod +r /etc/ceph/ceph.client.admin.keyring 2024-01-27T23:23:49.821 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:-- Logs begin at Sat 2024-01-27 23:13:52 UTC. -- 2024-01-27T23:23:50.100 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-27T23:23:50.961 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: 4.6.1 2024-01-27T23:23:50.971 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying podman|docker is present... 2024-01-27T23:23:50.971 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-27T23:23:50.994 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: 4.6.1 2024-01-27T23:23:51.004 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying lvm2 is present... 2024-01-27T23:23:51.005 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying time synchronization is in place... 2024-01-27T23:23:51.005 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled chrony.service 2024-01-27T23:23:51.012 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-27T23:23:51.013 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active chrony.service 2024-01-27T23:23:51.018 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-01-27T23:23:51.019 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-27T23:23:51.025 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: enabled 2024-01-27T23:23:51.025 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active chronyd.service 2024-01-27T23:23:51.031 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: active 2024-01-27T23:23:51.032 INFO:teuthology.orchestra.run.smithi121.stderr:Unit chronyd.service is enabled and running 2024-01-27T23:23:51.032 INFO:teuthology.orchestra.run.smithi121.stderr:Repeating the final host check... 2024-01-27T23:23:51.032 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman version --format {{.Client.Version}} 2024-01-27T23:23:51.052 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: 4.6.1 2024-01-27T23:23:51.060 INFO:teuthology.orchestra.run.smithi121.stderr:podman|docker (/bin/podman) is present 2024-01-27T23:23:51.060 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl is present 2024-01-27T23:23:51.060 INFO:teuthology.orchestra.run.smithi121.stderr:lvcreate is present 2024-01-27T23:23:51.060 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled chrony.service 2024-01-27T23:23:51.066 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to get unit file state for chrony.service: No such file or directory 2024-01-27T23:23:51.066 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active chrony.service 2024-01-27T23:23:51.072 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-01-27T23:23:51.072 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled chronyd.service 2024-01-27T23:23:51.078 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: enabled 2024-01-27T23:23:51.078 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active chronyd.service 2024-01-27T23:23:51.083 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: active 2024-01-27T23:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:Unit chronyd.service is enabled and running 2024-01-27T23:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:Host looks OK 2024-01-27T23:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:Cluster fsid: 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:23:51.084 INFO:teuthology.orchestra.run.smithi121.stderr:Acquiring lock 139728639085032 on /run/cephadm/2005e414-bd6b-11ee-95b2-87774f69a715.lock 2024-01-27T23:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:Lock 139728639085032 acquired on /run/cephadm/2005e414-bd6b-11ee-95b2-87774f69a715.lock 2024-01-27T23:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying IP 172.21.15.121 port 3300 ... 2024-01-27T23:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying IP 172.21.15.121 port 6789 ... 2024-01-27T23:23:51.085 INFO:teuthology.orchestra.run.smithi121.stderr:Base mon IP is 172.21.15.121, final addrv is [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-01-27T23:23:51.086 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /sbin/ip route ls 2024-01-27T23:23:51.088 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: default via 172.21.15.254 dev enp3s0f1 proto dhcp src 172.21.15.121 metric 100 2024-01-27T23:23:51.088 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 172.21.0.0/20 dev enp3s0f1 proto kernel scope link src 172.21.15.121 metric 100 2024-01-27T23:23:51.088 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /sbin/ip -6 route ls 2024-01-27T23:23:51.090 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: ::1 dev lo proto kernel metric 256 pref medium 2024-01-27T23:23:51.091 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: fe80::/64 dev enp3s0f1 proto kernel metric 256 pref medium 2024-01-27T23:23:51.091 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: default via fe80::327c:5e00:6487:71e0 dev enp3s0f1 proto ra metric 1024 expires 1799sec hoplimit 64 pref medium 2024-01-27T23:23:51.091 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /sbin/ip -6 addr ls 2024-01-27T23:23:51.093 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 1: lo: mtu 65536 state UNKNOWN qlen 1000 2024-01-27T23:23:51.094 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: inet6 ::1/128 scope host 2024-01-27T23:23:51.094 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-27T23:23:51.094 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: 5: enp3s0f1: mtu 1500 state UP qlen 1000 2024-01-27T23:23:51.094 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: inet6 fe80::ec4:7aff:fe8f:cba3/64 scope link 2024-01-27T23:23:51.094 INFO:teuthology.orchestra.run.smithi121.stderr:/sbin/ip: valid_lft forever preferred_lft forever 2024-01-27T23:23:51.094 INFO:teuthology.orchestra.run.smithi121.stderr:Mon IP 172.21.15.121 is in CIDR network 172.21.0.0/20 2024-01-27T23:23:51.095 INFO:teuthology.orchestra.run.smithi121.stderr:- internal network (--cluster-network) has not been provided, OSD replication will default to the public_network 2024-01-27T23:23:51.096 INFO:teuthology.orchestra.run.smithi121.stderr:Pulling container image docker.io/ceph/ceph:v16.2.4... 2024-01-27T23:23:51.096 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman pull docker.io/ceph/ceph:v16.2.4 2024-01-27T23:23:51.187 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: Trying to pull docker.io/ceph/ceph:v16.2.4... 2024-01-27T23:23:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: Getting image source signatures 2024-01-27T23:23:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: Copying blob sha256:551101690791055467b1840be133b2e6b029d263771bc6c46a96bb7acdacac39 2024-01-27T23:23:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: Copying blob sha256:7a0437f04f83f084b7ed68ad9c4a4947e12fc4e1b006b38129bac89114ec3621 2024-01-27T23:24:17.181 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: Copying config sha256:8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-27T23:24:17.336 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: Writing manifest to image destination 2024-01-27T23:24:17.678 INFO:teuthology.orchestra.run.smithi121.stderr:/bin/podman: 8d91d370c2b86c07de46146aba8d36718eaefa69b1880c77fa312fda6efd7d29 2024-01-27T23:24:17.686 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --version 2024-01-27T23:24:18.226 INFO:teuthology.orchestra.run.smithi121.stderr:ceph: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-27T23:24:18.607 INFO:teuthology.orchestra.run.smithi121.stderr:Ceph version: ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable) 2024-01-27T23:24:18.607 INFO:teuthology.orchestra.run.smithi121.stderr:Extracting ceph user uid/gid from container image... 2024-01-27T23:24:18.607 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 -c %u %g /var/lib/ceph 2024-01-27T23:24:18.896 INFO:teuthology.orchestra.run.smithi121.stderr:stat: 167 167 2024-01-27T23:24:19.208 INFO:teuthology.orchestra.run.smithi121.stderr:Creating initial keys... 2024-01-27T23:24:19.208 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-27T23:24:19.479 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-authtool: AQCjkLVlkc+xHBAAqHWrgZxwYQ2hYVc65UzwHQ== 2024-01-27T23:24:19.825 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-27T23:24:20.079 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-authtool: AQCkkLVlBQpbBBAAIm0kOkcCbshFQS66JaS15g== 2024-01-27T23:24:20.395 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-authtool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/ceph/ceph:v16.2.4 --gen-print-key 2024-01-27T23:24:20.637 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-authtool: AQCkkLVlKWPXJRAAjnhHB/lNV1b/SUyfRpzlPA== 2024-01-27T23:24:20.983 INFO:teuthology.orchestra.run.smithi121.stderr:Creating initial monmap... 2024-01-27T23:24:20.983 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/monmaptool --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /tmp/ceph-tmp6emhk1x1:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --create --clobber --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 --addv smithi121 [v2:172.21.15.121:3300,v1:172.21.15.121:6789] /tmp/monmap 2024-01-27T23:24:21.282 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-27T23:24:21.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: set fsid to 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:21.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: /usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-27T23:24:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:monmaptool for smithi121 [v2:172.21.15.121:3300,v1:172.21.15.121:6789] on /usr/bin/monmaptool: monmap file /tmp/monmap 2024-01-27T23:24:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: set fsid to 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/monmaptool: writing epoch 0 to /tmp/monmap (1 monitors) 2024-01-27T23:24:21.617 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:24:21.617 INFO:teuthology.orchestra.run.smithi121.stderr:Creating mon... 2024-01-27T23:24:21.618 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph-mon --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/mon.smithi121:/var/lib/ceph/mon/ceph-smithi121:z -v /tmp/ceph-tmpkqhvn8hf:/tmp/keyring:z -v /tmp/ceph-tmp6emhk1x1:/tmp/monmap:z docker.io/ceph/ceph:v16.2.4 --mkfs -i smithi121 --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -c /dev/null --monmap /tmp/monmap --keyring /tmp/keyring --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true 2024-01-27T23:24:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.880+0000 7f407c209700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-27T23:24:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.880+0000 7f407c209700 1 imported monmap: 2024-01-27T23:24:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: epoch 0 2024-01-27T23:24:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: fsid 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: last_changed 2024-01-27T23:24:21.255875+0000 2024-01-27T23:24:21.904 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: created 2024-01-27T23:24:21.255875+0000 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: min_mon_release 0 (unknown) 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: election_strategy: 1 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 0: [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon.smithi121 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.880+0000 7f407c209700 0 /usr/bin/ceph-mon: set fsid to 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: RocksDB version: 6.8.1 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Compile date May 13 2021 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: DB SUMMARY 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi121/store.db dir, Total Num: 0, files: 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi121/store.db: 2024-01-27T23:24:21.905 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.error_if_exists: 0 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.create_if_missing: 1 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.paranoid_checks: 1 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.env: 0x555c95c331c0 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.fs: Posix File System 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.info_log: 0x555c9780cec0 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.statistics: (nil) 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.use_fsync: 0 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_log_file_size: 0 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-27T23:24:21.906 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.allow_fallocate: 1 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.use_direct_reads: 0 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.db_log_dir: 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi121/store.db 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_subcompactions: 1 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_background_flushes: -1 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-27T23:24:21.907 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.write_buffer_manager: 0x555c97813530 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-27T23:24:21.908 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.unordered_write: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.row_cache: None 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.wal_filter: None 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.preserve_deletes: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.two_write_queues: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.atomic_flush: 0 2024-01-27T23:24:21.909 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.log_readahead_size: 0 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_background_jobs: 2 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_background_compactions: -1 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-27T23:24:21.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_open_files: -1 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Compression algorithms supported: 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kZSTD supported: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kXpressCompression supported: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kLZ4Compression supported: 1 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kBZip2Compression supported: 0 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kZlibCompression supported: 1 2024-01-27T23:24:21.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: kSnappyCompression supported: 1 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: [db_impl/db_impl_open.cc:273] Creating manifest 1 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000001 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.merge_operator: 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_filter: None 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-27T23:24:21.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x555c97724120) 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks: 1 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: pin_top_level_index_and_filter: 1 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: index_type: 0 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: data_block_index_type: 0 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: index_shortening: 1 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: data_block_hash_table_util_ratio: 0.750000 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: hash_index_allow_collision: 1 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: checksum: 1 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: no_block_cache: 0 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache: 0x555c9775ad10 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache_name: BinnedLRUCache 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache_options: 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: capacity : 536870912 2024-01-27T23:24:21.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: num_shard_bits : 4 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: strict_capacity_limit : 0 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: high_pri_pool_ratio: 0.000 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_cache_compressed: (nil) 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: persistent_cache: (nil) 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_size: 4096 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_size_deviation: 10 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_restart_interval: 16 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: index_block_restart_interval: 1 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: metadata_block_size: 4096 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: partition_filters: 0 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: use_delta_encoding: 1 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: whole_key_filtering: 1 2024-01-27T23:24:21.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: verify_compression: 0 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: read_amp_bytes_per_bit: 0 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: format_version: 2 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: enable_index_compression: 1 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: block_align: 0 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compression: NoCompression 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.num_levels: 7 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-27T23:24:21.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-27T23:24:21.916 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-27T23:24:21.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.table_properties_collectors: 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.inplace_update_support: 0 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-27T23:24:21.918 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.bloom_locality: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.max_successive_merges: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.ttl: 2592000 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.881+0000 7f407c209700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.882+0000 7f407c209700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000001 succeeded,manifest_file_number is 1, next_file_number is 3, last_sequence is 0, log_number is 0,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.882+0000 7f407c209700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 0 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.882+0000 7f407c209700 4 rocksdb: DB pointer 0x555c9781f800 2024-01-27T23:24:21.919 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.883+0000 7f40650a2700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.883+0000 7f40650a2700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** DB Stats ** 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:21.920 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Sum 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** Compaction Stats [default] ** 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Uptime(secs): 0.0 total, 0.0 interval 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Flush(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Total Files): cumulative 0, interval 0 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(L0 Files): cumulative 0, interval 0 2024-01-27T23:24:21.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: AddFile(Keys): cumulative 0, interval 0 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Cumulative compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: ** File Read Latency Histogram By Level [default] ** 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.884+0000 7f407c209700 4 rocksdb: [db_impl/db_impl.cc:397] Shutdown: canceling all background work 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.884+0000 7f407c209700 4 rocksdb: [db_impl/db_impl.cc:573] Shutdown complete 2024-01-27T23:24:21.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph-mon: debug 2024-01-27T23:24:21.884+0000 7f407c209700 0 /usr/bin/ceph-mon: created monfs at /var/lib/ceph/mon/ceph-smithi121 for mon.smithi121 2024-01-27T23:24:22.231 INFO:teuthology.orchestra.run.smithi121.stderr:create mon.smithi121 on 2024-01-27T23:24:22.231 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: install -d -m0770 -o 167 -g 167 /var/run/ceph/2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:22.236 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl disable ceph.target 2024-01-27T23:24:22.242 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Removed /etc/systemd/system/multi-user.target.wants/ceph.target. 2024-01-27T23:24:22.324 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl enable ceph.target 2024-01-27T23:24:22.331 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph.target → /etc/systemd/system/ceph.target. 2024-01-27T23:24:22.413 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl start ceph.target 2024-01-27T23:24:22.420 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl enable ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target 2024-01-27T23:24:22.425 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/multi-user.target.wants/ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target → /etc/systemd/system/ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target. 2024-01-27T23:24:22.426 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/ceph.target.wants/ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target → /etc/systemd/system/ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target. 2024-01-27T23:24:22.508 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl start ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target 2024-01-27T23:24:22.515 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl daemon-reload 2024-01-27T23:24:22.600 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121 2024-01-27T23:24:22.624 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl reset-failed ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121 2024-01-27T23:24:22.629 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to reset failed state of unit ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121.service: Unit ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121.service not loaded. 2024-01-27T23:24:22.630 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl enable ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121 2024-01-27T23:24:22.636 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target.wants/ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121.service → /etc/systemd/system/ceph-2005e414-bd6b-11ee-95b2-87774f69a715@.service. 2024-01-27T23:24:22.722 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl start ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121 2024-01-27T23:24:23.015 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-27T23:24:23.023 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: disabled 2024-01-27T23:24:23.024 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active firewalld.service 2024-01-27T23:24:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-01-27T23:24:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld.service is not enabled 2024-01-27T23:24:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to enable service . firewalld.service is not available 2024-01-27T23:24:23.031 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-27T23:24:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mon to start... 2024-01-27T23:24:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mon... 2024-01-27T23:24:23.032 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/mon.smithi121:/var/lib/ceph/mon/ceph-smithi121:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status 2024-01-27T23:24:23.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:23 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: cluster 2024-01-27T23:24:23.041965+0000 mon.smithi121 (mon.0) 1 : cluster [INF] mon.smithi121 is new leader, mons smithi121 in quorum (ranks 0) 2024-01-27T23:24:23.415 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.425+0000 7fb5acf9f700 1 Processor -- start 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.425+0000 7fb5acf9f700 1 -- start start 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.426+0000 7fb5acf9f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a80fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.426+0000 7fb5acf9f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a80fb490 con 0x7fb5a80fab80 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.426+0000 7fb5a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a80fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.426+0000 7fb5a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a80fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42674/0 (socket says 172.21.15.121:42674) 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.426+0000 7fb5a659c700 1 -- 172.21.15.121:0/4040834310 learned_addr learned my addr 172.21.15.121:0/4040834310 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.426+0000 7fb5a659c700 1 -- 172.21.15.121:0/4040834310 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb5a80fb5d0 con 0x7fb5a80fab80 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.426+0000 7fb5a659c700 1 --2- 172.21.15.121:0/4040834310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a80fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb59c00b5f0 tx=0x7fb59c006ee0).ready entity=mon.0 client_cookie=3004cb29d559ecb2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5a559a700 1 -- 172.21.15.121:0/4040834310 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb59c00d600 con 0x7fb5a80fab80 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5a559a700 1 -- 172.21.15.121:0/4040834310 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c00d760 con 0x7fb5a80fab80 2024-01-27T23:24:23.417 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5a559a700 1 -- 172.21.15.121:0/4040834310 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb59c00da30 con 0x7fb5a80fab80 2024-01-27T23:24:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5acf9f700 1 -- 172.21.15.121:0/4040834310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 msgr2=0x7fb5a80fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5acf9f700 1 --2- 172.21.15.121:0/4040834310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a80fafa0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fb59c00b5f0 tx=0x7fb59c006ee0).stop 2024-01-27T23:24:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5acf9f700 1 -- 172.21.15.121:0/4040834310 shutdown_connections 2024-01-27T23:24:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5acf9f700 1 --2- 172.21.15.121:0/4040834310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a80fafa0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5acf9f700 1 -- 172.21.15.121:0/4040834310 >> 172.21.15.121:0/4040834310 conn(0x7fb5a80f63c0 msgr2=0x7fb5a80f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:23.419 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5acf9f700 1 -- 172.21.15.121:0/4040834310 shutdown_connections 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.427+0000 7fb5acf9f700 1 -- 172.21.15.121:0/4040834310 wait complete. 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5acf9f700 1 Processor -- start 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5acf9f700 1 -- start start 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5acf9f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a8109150 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5acf9f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb5a8109640 con 0x7fb5a80fab80 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a8109150 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5a659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a8109150 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42680/0 (socket says 172.21.15.121:42680) 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5a659c700 1 -- 172.21.15.121:0/3124712237 learned_addr learned my addr 172.21.15.121:0/3124712237 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.428+0000 7fb5a659c700 1 -- 172.21.15.121:0/3124712237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb59c00b200 con 0x7fb5a80fab80 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb5a659c700 1 --2- 172.21.15.121:0/3124712237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a8109150 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb59c006c70 tx=0x7fb59c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb596ffd700 1 -- 172.21.15.121:0/3124712237 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb59c011b40 con 0x7fb5a80fab80 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb596ffd700 1 -- 172.21.15.121:0/3124712237 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7fb59c011ca0 con 0x7fb5a80fab80 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb5a8109840 con 0x7fb5a80fab80 2024-01-27T23:24:23.420 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb596ffd700 1 -- 172.21.15.121:0/3124712237 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb59c01d430 con 0x7fb5a80fab80 2024-01-27T23:24:23.421 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb5a8109c50 con 0x7fb5a80fab80 2024-01-27T23:24:23.421 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb596ffd700 1 -- 172.21.15.121:0/3124712237 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7fb59c00f070 con 0x7fb5a80fab80 2024-01-27T23:24:23.421 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.429+0000 7fb596ffd700 1 -- 172.21.15.121:0/3124712237 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fb59c027460 con 0x7fb5a80fab80 2024-01-27T23:24:23.421 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.430+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb5a8045ad0 con 0x7fb5a80fab80 2024-01-27T23:24:23.421 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.432+0000 7fb596ffd700 1 -- 172.21.15.121:0/3124712237 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7fb59c016070 con 0x7fb5a80fab80 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.498+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status"} v 0) v1 -- 0x7fb5a8102b50 con 0x7fb5a80fab80 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.498+0000 7fb596ffd700 1 -- 172.21.15.121:0/3124712237 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status"}]=0 v0) v1 ==== 54+0+324 (secure 0 0 0) 0x7fb59c01d640 con 0x7fb5a80fab80 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: cluster: 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: id: 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: health: HEALTH_OK 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: services: 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon: 1 daemons, quorum smithi121 (age 0.44226s) 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mgr: no daemons active 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd: 0 osds: 0 up, 0 in 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.490 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: data: 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: pools: 0 pools, 0 pgs 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: objects: 0 objects, 0 B 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: usage: 0 B used, 0 B / 0 B avail 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: pgs: 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.499+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 msgr2=0x7fb5a8109150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.499+0000 7fb5acf9f700 1 --2- 172.21.15.121:0/3124712237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a8109150 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fb59c006c70 tx=0x7fb59c016000).stop 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.499+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 shutdown_connections 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.499+0000 7fb5acf9f700 1 --2- 172.21.15.121:0/3124712237 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb5a80fab80 0x7fb5a8109150 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.499+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 >> 172.21.15.121:0/3124712237 conn(0x7fb5a80f63c0 msgr2=0x7fb5a80fd900 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.499+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 shutdown_connections 2024-01-27T23:24:23.491 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:23.499+0000 7fb5acf9f700 1 -- 172.21.15.121:0/3124712237 wait complete. 2024-01-27T23:24:23.839 INFO:teuthology.orchestra.run.smithi121.stderr:mon is available 2024-01-27T23:24:23.839 INFO:teuthology.orchestra.run.smithi121.stderr:Assimilating anything we can from ceph.conf... 2024-01-27T23:24:23.839 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z -v /var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/mon.smithi121:/var/lib/ceph/mon/ceph-smithi121:z docker.io/ceph/ceph:v16.2.4 config assimilate-conf -i /var/lib/ceph/mon/ceph-smithi121/config 2024-01-27T23:24:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.221+0000 7f9cb198d700 1 Processor -- start 2024-01-27T23:24:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.221+0000 7f9cb198d700 1 -- start start 2024-01-27T23:24:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.221+0000 7f9cb198d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0fc0b0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.221+0000 7f9cb198d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac0fc5a0 con 0x7f9cac0fbc90 2024-01-27T23:24:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9caaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0fc0b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9caaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0fc0b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42684/0 (socket says 172.21.15.121:42684) 2024-01-27T23:24:24.212 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9caaffd700 1 -- 172.21.15.121:0/3814845045 learned_addr learned my addr 172.21.15.121:0/3814845045 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9caaffd700 1 -- 172.21.15.121:0/3814845045 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cac0fc6e0 con 0x7f9cac0fbc90 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9caaffd700 1 --2- 172.21.15.121:0/3814845045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0fc0b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f9c9c00b5f0 tx=0x7f9c9c006ee0).ready entity=mon.0 client_cookie=c8a507527276b3a6 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9ca9ffb700 1 -- 172.21.15.121:0/3814845045 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c00d600 con 0x7f9cac0fbc90 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9ca9ffb700 1 -- 172.21.15.121:0/3814845045 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c00d760 con 0x7f9cac0fbc90 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.222+0000 7f9ca9ffb700 1 -- 172.21.15.121:0/3814845045 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c00da30 con 0x7f9cac0fbc90 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 -- 172.21.15.121:0/3814845045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 msgr2=0x7f9cac0fc0b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 --2- 172.21.15.121:0/3814845045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0fc0b0 secure :-1 s=READY pgs=3 cs=0 l=1 rev1=1 rx=0x7f9c9c00b5f0 tx=0x7f9c9c006ee0).stop 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 -- 172.21.15.121:0/3814845045 shutdown_connections 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 --2- 172.21.15.121:0/3814845045 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0fc0b0 unknown :-1 s=CLOSED pgs=3 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 -- 172.21.15.121:0/3814845045 >> 172.21.15.121:0/3814845045 conn(0x7f9cac0f74b0 msgr2=0x7f9cac0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 -- 172.21.15.121:0/3814845045 shutdown_connections 2024-01-27T23:24:24.215 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 -- 172.21.15.121:0/3814845045 wait complete. 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.223+0000 7f9cb198d700 1 Processor -- start 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9cb198d700 1 -- start start 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9cb198d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0725e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9cb198d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9cac070a60 con 0x7f9cac0fbc90 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9caaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0725e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9caaffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0725e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42690/0 (socket says 172.21.15.121:42690) 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9caaffd700 1 -- 172.21.15.121:0/2805705540 learned_addr learned my addr 172.21.15.121:0/2805705540 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9caaffd700 1 -- 172.21.15.121:0/2805705540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9c9c00b200 con 0x7f9cac0fbc90 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9caaffd700 1 --2- 172.21.15.121:0/2805705540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0725e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9c9c006c70 tx=0x7f9c9c016000).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c011b40 con 0x7f9cac0fbc90 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(0 keys) v1 ==== 4+0+0 (secure 0 0 0) 0x7f9c9c011ca0 con 0x7f9cac0fbc90 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9cac070c00 con 0x7f9cac0fbc90 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9c9c01d430 con 0x7f9cac0fbc90 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.224+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9cac071010 con 0x7f9cac0fbc90 2024-01-27T23:24:24.216 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.225+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f9c9c00f070 con 0x7f9cac0fbc90 2024-01-27T23:24:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.225+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9c9c027460 con 0x7f9cac0fbc90 2024-01-27T23:24:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.226+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9cac045ad0 con 0x7f9cac0fbc90 2024-01-27T23:24:24.217 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.228+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f9c9c016070 con 0x7f9cac0fbc90 2024-01-27T23:24:24.281 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.292+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config assimilate-conf"} v 0) v1 -- 0x7f9cac0fcfd0 con 0x7f9cac0fbc90 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.295+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 7 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9c9c01d590 con 0x7f9cac0fbc90 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.295+0000 7f9ca37fe700 1 -- 172.21.15.121:0/2805705540 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "config assimilate-conf"}]=0 v2) v1 ==== 70+0+466 (secure 0 0 0) 0x7f9c9c0276f0 con 0x7f9cac0fbc90 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [global] 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: fsid = 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_host = [v2:172.21.15.121:3300,v1:172.21.15.121:6789] 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_allow_pg_remap = true 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_allow_primary_affinity = true 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_warn_on_no_sortbitwise = false 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_crush_chooseleaf_type = 0 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [mgr] 2024-01-27T23:24:24.286 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mgr/telemetry/nag = false 2024-01-27T23:24:24.287 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.287 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: [osd] 2024-01-27T23:24:24.288 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_backfillfull_ratio = 0.85 2024-01-27T23:24:24.288 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_full_ratio = 0.9 2024-01-27T23:24:24.288 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: mon_osd_nearfull_ratio = 0.8 2024-01-27T23:24:24.288 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_map_max_advance = 10 2024-01-27T23:24:24.288 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_memory_target_autotune = true 2024-01-27T23:24:24.288 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: osd_sloppy_crc = true 2024-01-27T23:24:24.288 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.296+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 msgr2=0x7f9cac0725e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:24.289 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.296+0000 7f9cb198d700 1 --2- 172.21.15.121:0/2805705540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0725e0 secure :-1 s=READY pgs=4 cs=0 l=1 rev1=1 rx=0x7f9c9c006c70 tx=0x7f9c9c016000).stop 2024-01-27T23:24:24.289 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.296+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 shutdown_connections 2024-01-27T23:24:24.289 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.296+0000 7f9cb198d700 1 --2- 172.21.15.121:0/2805705540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9cac0fbc90 0x7f9cac0725e0 unknown :-1 s=CLOSED pgs=4 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:24.289 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.296+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 >> 172.21.15.121:0/2805705540 conn(0x7f9cac0f74b0 msgr2=0x7f9cac0fea40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:24.289 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.296+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 shutdown_connections 2024-01-27T23:24:24.289 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:24.296+0000 7f9cb198d700 1 -- 172.21.15.121:0/2805705540 wait complete. 2024-01-27T23:24:24.308 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: cluster 2024-01-27T23:24:23.057767+0000 mon.smithi121 (mon.0) 2 : cluster [INF] mon.smithi121 is new leader, mons smithi121 in quorum (ranks 0) 2024-01-27T23:24:24.309 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: cluster 2024-01-27T23:24:23.057999+0000 mon.smithi121 (mon.0) 3 : cluster [DBG] monmap e1: 1 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0]} 2024-01-27T23:24:24.309 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: cluster 2024-01-27T23:24:23.060822+0000 mon.smithi121 (mon.0) 4 : cluster [DBG] fsmap 2024-01-27T23:24:24.309 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: cluster 2024-01-27T23:24:23.062736+0000 mon.smithi121 (mon.0) 5 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-27T23:24:24.309 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: cluster 2024-01-27T23:24:23.063158+0000 mon.smithi121 (mon.0) 6 : cluster [DBG] mgrmap e1: no daemons active 2024-01-27T23:24:24.309 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: audit 2024-01-27T23:24:23.499975+0000 mon.smithi121 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.121:0/3124712237' entity='client.admin' cmd=[{"prefix": "status"}]: dispatch 2024-01-27T23:24:24.638 INFO:teuthology.orchestra.run.smithi121.stderr:Generating new minimal ceph.conf... 2024-01-27T23:24:24.638 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z -v /var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/mon.smithi121:/var/lib/ceph/mon/ceph-smithi121:z docker.io/ceph/ceph:v16.2.4 config generate-minimal-conf -o /var/lib/ceph/mon/ceph-smithi121/config 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.010+0000 7f657f01a700 1 Processor -- start 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.010+0000 7f657f01a700 1 -- start start 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.011+0000 7f657f01a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f65780fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.011+0000 7f657f01a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f65780fee10 con 0x7f65780703f0 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.011+0000 7f657cdb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f65780fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.011+0000 7f657cdb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f65780fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42700/0 (socket says 172.21.15.121:42700) 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.011+0000 7f657cdb6700 1 -- 172.21.15.121:0/1004723464 learned_addr learned my addr 172.21.15.121:0/1004723464 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:25.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.011+0000 7f657cdb6700 1 -- 172.21.15.121:0/1004723464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f65780fef50 con 0x7f65780703f0 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.011+0000 7f657cdb6700 1 --2- 172.21.15.121:0/1004723464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f65780fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f656c00b5f0 tx=0x7f656c006ee0).ready entity=mon.0 client_cookie=e0fbf3a9f687e468 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f65777fe700 1 -- 172.21.15.121:0/1004723464 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f656c00d600 con 0x7f65780703f0 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f65777fe700 1 -- 172.21.15.121:0/1004723464 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f656c00dbd0 con 0x7f65780703f0 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f65777fe700 1 -- 172.21.15.121:0/1004723464 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f656c011c00 con 0x7f65780703f0 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f657f01a700 1 -- 172.21.15.121:0/1004723464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 msgr2=0x7f65780fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f657f01a700 1 --2- 172.21.15.121:0/1004723464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f65780fe920 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f656c00b5f0 tx=0x7f656c006ee0).stop 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f657f01a700 1 -- 172.21.15.121:0/1004723464 shutdown_connections 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f657f01a700 1 --2- 172.21.15.121:0/1004723464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f65780fe920 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.012+0000 7f657f01a700 1 -- 172.21.15.121:0/1004723464 >> 172.21.15.121:0/1004723464 conn(0x7f65780f63c0 msgr2=0x7f65780f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657f01a700 1 -- 172.21.15.121:0/1004723464 shutdown_connections 2024-01-27T23:24:25.003 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657f01a700 1 -- 172.21.15.121:0/1004723464 wait complete. 2024-01-27T23:24:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657f01a700 1 Processor -- start 2024-01-27T23:24:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657f01a700 1 -- start start 2024-01-27T23:24:25.004 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657f01a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f657810a9c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657f01a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f657810aeb0 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657cdb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f657810a9c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657cdb6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f657810a9c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42702/0 (socket says 172.21.15.121:42702) 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.013+0000 7f657cdb6700 1 -- 172.21.15.121:0/897702810 learned_addr learned my addr 172.21.15.121:0/897702810 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f657cdb6700 1 -- 172.21.15.121:0/897702810 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f656c00b200 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f657cdb6700 1 --2- 172.21.15.121:0/897702810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f657810a9c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f656c00b190 tx=0x7f656c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f65757fa700 1 -- 172.21.15.121:0/897702810 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f656c01b410 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f65757fa700 1 -- 172.21.15.121:0/897702810 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f656c01b9e0 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f657810b0b0 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f65757fa700 1 -- 172.21.15.121:0/897702810 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f656c01a880 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f657810b4c0 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f65757fa700 1 -- 172.21.15.121:0/897702810 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f656c00f070 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.014+0000 7f65757fa700 1 -- 172.21.15.121:0/897702810 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f656c025c90 con 0x7f65780703f0 2024-01-27T23:24:25.005 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.015+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6578045ad0 con 0x7f65780703f0 2024-01-27T23:24:25.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.017+0000 7f65757fa700 1 -- 172.21.15.121:0/897702810 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f656c016070 con 0x7f65780703f0 2024-01-27T23:24:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.082+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config generate-minimal-conf"} v 0) v1 -- 0x7f6578104890 con 0x7f65780703f0 2024-01-27T23:24:25.071 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.082+0000 7f65757fa700 1 -- 172.21.15.121:0/897702810 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config generate-minimal-conf"}]=0 v2) v1 ==== 76+0+177 (secure 0 0 0) 0x7f656c01ae90 con 0x7f65780703f0 2024-01-27T23:24:25.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.082+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 msgr2=0x7f657810a9c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:25.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.082+0000 7f657f01a700 1 --2- 172.21.15.121:0/897702810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f657810a9c0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f656c00b190 tx=0x7f656c006ee0).stop 2024-01-27T23:24:25.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.083+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 shutdown_connections 2024-01-27T23:24:25.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.083+0000 7f657f01a700 1 --2- 172.21.15.121:0/897702810 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f65780703f0 0x7f657810a9c0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:25.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.083+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 >> 172.21.15.121:0/897702810 conn(0x7f65780f63c0 msgr2=0x7f65780f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:25.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.083+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 shutdown_connections 2024-01-27T23:24:25.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:25.083+0000 7f657f01a700 1 -- 172.21.15.121:0/897702810 wait complete. 2024-01-27T23:24:25.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: audit 2024-01-27T23:24:24.293528+0000 mon.smithi121 (mon.0) 8 : audit [INF] from='client.? 172.21.15.121:0/2805705540' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-27T23:24:25.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: audit 2024-01-27T23:24:24.296717+0000 mon.smithi121 (mon.0) 9 : audit [INF] from='client.? 172.21.15.121:0/2805705540' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-27T23:24:25.422 INFO:teuthology.orchestra.run.smithi121.stderr:Restarting the monitor... 2024-01-27T23:24:25.422 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl restart ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121 2024-01-27T23:24:25.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 systemd[1]: Stopping Ceph mon.smithi121 for 2005e414-bd6b-11ee-95b2-87774f69a715... 2024-01-27T23:24:25.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: debug 2024-01-27T23:24:25.513+0000 7f66e2ed5700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi121 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-27T23:24:25.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38373]: debug 2024-01-27T23:24:25.513+0000 7f66e2ed5700 -1 mon.smithi121@0(leader) e1 *** Got Signal Terminated *** 2024-01-27T23:24:26.154 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 podman[38701]: ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121 2024-01-27T23:24:26.154 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 systemd[1]: ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121.service: Succeeded. 2024-01-27T23:24:26.155 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 systemd[1]: Stopped Ceph mon.smithi121 for 2005e414-bd6b-11ee-95b2-87774f69a715. 2024-01-27T23:24:26.155 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:25 smithi121 systemd[1]: Starting Ceph mon.smithi121 for 2005e414-bd6b-11ee-95b2-87774f69a715... 2024-01-27T23:24:26.232 INFO:teuthology.orchestra.run.smithi121.stderr:Setting mon public_network to 172.21.0.0/20 2024-01-27T23:24:26.232 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mon public_network 172.21.0.0/20 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.231+0000 7f852852b700 0 set uid:gid to 167:167 (ceph:ceph) 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.231+0000 7f852852b700 0 ceph version 16.2.4 (3cbe25cde3cfa028984618ad32de9edc4c1eaed0) pacific (stable), process ceph-mon, pid 7 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.231+0000 7f852852b700 0 pidfile_write: ignore empty --pid-file 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 0 load: jerasure load: lrc load: isa 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: RocksDB version: 6.8.1 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Git sha rocksdb_build_git_sha:@0@ 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Compile date May 13 2021 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: DB SUMMARY 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: CURRENT file: CURRENT 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: IDENTITY file: IDENTITY 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: MANIFEST file: MANIFEST-000005 size: 131 Bytes 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: SST files in /var/lib/ceph/mon/ceph-smithi121/store.db dir, Total Num: 1, files: 000004.sst 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Write Ahead Log file in /var/lib/ceph/mon/ceph-smithi121/store.db: 000006.log size: 92938 ; 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.error_if_exists: 0 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.create_if_missing: 0 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.paranoid_checks: 1 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.env: 0x5572d4a021c0 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.fs: Posix File System 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.info_log: 0x5572d69fb960 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_file_opening_threads: 16 2024-01-27T23:24:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.statistics: (nil) 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.use_fsync: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_log_file_size: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_manifest_file_size: 1073741824 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.log_file_time_to_roll: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.keep_log_file_num: 1000 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.recycle_log_file_num: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.allow_fallocate: 1 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.allow_mmap_reads: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.allow_mmap_writes: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.use_direct_reads: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.use_direct_io_for_flush_and_compaction: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.create_missing_column_families: 0 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.db_log_dir: 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.wal_dir: /var/lib/ceph/mon/ceph-smithi121/store.db 2024-01-27T23:24:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.table_cache_numshardbits: 6 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_subcompactions: 1 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_background_flushes: -1 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.WAL_ttl_seconds: 0 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.WAL_size_limit_MB: 0 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_write_batch_group_size_bytes: 1048576 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.manifest_preallocation_size: 4194304 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.is_fd_close_on_exec: 1 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.advise_random_on_open: 1 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.db_write_buffer_size: 0 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.write_buffer_manager: 0x5572d6a01fb0 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.access_hint_on_compaction_start: 1 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.new_table_reader_for_compaction_inputs: 0 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.random_access_max_buffer_size: 1048576 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.use_adaptive_mutex: 0 2024-01-27T23:24:26.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.rate_limiter: (nil) 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.sst_file_manager.rate_bytes_per_sec: 0 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.wal_recovery_mode: 2 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.enable_thread_tracking: 0 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.enable_pipelined_write: 0 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.unordered_write: 0 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.allow_concurrent_memtable_write: 1 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.enable_write_thread_adaptive_yield: 1 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.write_thread_max_yield_usec: 100 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.write_thread_slow_yield_usec: 3 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.row_cache: None 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.wal_filter: None 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.avoid_flush_during_recovery: 0 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.allow_ingest_behind: 0 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 bash[38772]: c76700eeb4bf09de26dffbce45fff2bb92ac4e1c34d230f9796a650bb866877e 2024-01-27T23:24:26.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.preserve_deletes: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.two_write_queues: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.manual_wal_flush: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.atomic_flush: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.avoid_unnecessary_blocking_io: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.persist_stats_to_disk: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.write_dbid_to_manifest: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.log_readahead_size: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.sst_file_checksum_func: Unknown 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_background_jobs: 2 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_background_compactions: -1 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.avoid_flush_during_shutdown: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.writable_file_max_buffer_size: 1048576 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.delayed_write_rate : 16777216 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_total_wal_size: 0 2024-01-27T23:24:26.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.delete_obsolete_files_period_micros: 21600000000 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.stats_dump_period_sec: 600 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.stats_persist_period_sec: 600 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.stats_history_buffer_size: 1048576 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_open_files: -1 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bytes_per_sync: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.wal_bytes_per_sync: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.strict_bytes_per_sync: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_readahead_size: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Compression algorithms supported: 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kZSTDNotFinalCompression supported: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kZSTD supported: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kXpressCompression supported: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kLZ4HCCompression supported: 1 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kLZ4Compression supported: 1 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kBZip2Compression supported: 0 2024-01-27T23:24:26.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kZlibCompression supported: 1 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: kSnappyCompression supported: 1 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Fast CRC32 supported: Supported on x86 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: [version_set.cc:4413] Recovering from manifest file: /var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000005 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: [column_family.cc:552] --------------- Options for column family [default]: 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.comparator: leveldb.BytewiseComparator 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.merge_operator: 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_filter: None 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_filter_factory: None 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.memtable_factory: SkipListFactory 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.table_factory: BlockBasedTable 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: table_factory options: flush_block_policy_factory: FlushBlockBySizePolicyFactory (0x5572d6912128) 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cache_index_and_filter_blocks: 1 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cache_index_and_filter_blocks_with_high_priority: 0 2024-01-27T23:24:26.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: pin_l0_filter_and_index_blocks_in_cache: 0 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: pin_top_level_index_and_filter: 1 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: index_type: 0 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: data_block_index_type: 0 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: index_shortening: 1 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: data_block_hash_table_util_ratio: 0.750000 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: hash_index_allow_collision: 1 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: checksum: 1 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: no_block_cache: 0 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_cache: 0x5572d6948f10 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_cache_name: BinnedLRUCache 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_cache_options: 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: capacity : 536870912 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: num_shard_bits : 4 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: strict_capacity_limit : 0 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: high_pri_pool_ratio: 0.000 2024-01-27T23:24:26.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_cache_compressed: (nil) 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: persistent_cache: (nil) 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_size: 4096 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_size_deviation: 10 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_restart_interval: 16 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: index_block_restart_interval: 1 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: metadata_block_size: 4096 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: partition_filters: 0 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: use_delta_encoding: 1 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: filter_policy: rocksdb.BuiltinBloomFilter 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: whole_key_filtering: 1 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: verify_compression: 0 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: read_amp_bytes_per_bit: 0 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: format_version: 2 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: enable_index_compression: 1 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: block_align: 0 2024-01-27T23:24:26.597 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.write_buffer_size: 33554432 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_write_buffer_number: 2 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compression: NoCompression 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bottommost_compression: Disabled 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.prefix_extractor: nullptr 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.memtable_insert_with_hint_prefix_extractor: nullptr 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.num_levels: 7 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.min_write_buffer_number_to_merge: 1 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_write_buffer_number_to_maintain: 0 2024-01-27T23:24:26.598 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_write_buffer_size_to_maintain: 0 2024-01-27T23:24:26.599 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bottommost_compression_opts.window_bits: -14 2024-01-27T23:24:26.599 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bottommost_compression_opts.level: 32767 2024-01-27T23:24:26.599 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bottommost_compression_opts.strategy: 0 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bottommost_compression_opts.max_dict_bytes: 0 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bottommost_compression_opts.zstd_max_train_bytes: 0 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bottommost_compression_opts.enabled: false 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compression_opts.window_bits: -14 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compression_opts.level: 32767 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compression_opts.strategy: 0 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compression_opts.max_dict_bytes: 0 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compression_opts.zstd_max_train_bytes: 0 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compression_opts.enabled: false 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.level0_file_num_compaction_trigger: 4 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.level0_slowdown_writes_trigger: 20 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.level0_stop_writes_trigger: 36 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.target_file_size_base: 67108864 2024-01-27T23:24:26.600 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.target_file_size_multiplier: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_base: 268435456 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.level_compaction_dynamic_level_bytes: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier: 10.000000 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[0]: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[1]: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[2]: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[3]: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[4]: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[5]: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_bytes_for_level_multiplier_addtl[6]: 1 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_sequential_skip_in_iterations: 8 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_compaction_bytes: 1677721600 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.arena_block_size: 4194304 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.soft_pending_compaction_bytes_limit: 68719476736 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.hard_pending_compaction_bytes_limit: 274877906944 2024-01-27T23:24:26.601 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.rate_limit_delay_max_milliseconds: 100 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.disable_auto_compactions: 0 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_style: kCompactionStyleLevel 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_pri: kMinOverlappingRatio 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_universal.size_ratio: 1 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_universal.min_merge_width: 2 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_universal.max_merge_width: 4294967295 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_universal.max_size_amplification_percent: 200 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_universal.compression_size_percent: -1 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_universal.stop_style: kCompactionStopStyleTotalSize 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_fifo.max_table_files_size: 1073741824 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.compaction_options_fifo.allow_compaction: 0 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.table_properties_collectors: 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.inplace_update_support: 0 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.inplace_update_num_locks: 10000 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.memtable_prefix_bloom_size_ratio: 0.000000 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.memtable_whole_key_filtering: 0 2024-01-27T23:24:26.602 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.memtable_huge_page_size: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.bloom_locality: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.max_successive_merges: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.optimize_filters_for_hits: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.paranoid_file_checks: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.force_consistency_checks: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.report_bg_io_stats: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.ttl: 2592000 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: Options.periodic_compaction_seconds: 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.236+0000 7f852852b700 4 rocksdb: [version_set.cc:4568] Recovered from manifest file:/var/lib/ceph/mon/ceph-smithi121/store.db/MANIFEST-000005 succeeded,manifest_file_number is 5, next_file_number is 7, last_sequence is 5, log_number is 4,prev_log_number is 0,max_column_family is 0,min_log_number_to_keep is 0 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.237+0000 7f852852b700 4 rocksdb: [version_set.cc:4577] Column family [default] (ID 0), log number is 4 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.237+0000 7f852852b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1706397866238259, "job": 1, "event": "recovery_started", "log_files": [6]} 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.237+0000 7f852852b700 4 rocksdb: [db_impl/db_impl_open.cc:760] Recovering log #6 mode 2 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.237+0000 7f852852b700 3 rocksdb: [le/block_based/filter_policy.cc:584] Using legacy Bloom filter with high (20) bits/key. Dramatic filter space and/or accuracy improvement is available with format_version>=5. 2024-01-27T23:24:26.603 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.237+0000 7f852852b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1706397866239084, "cf_name": "default", "job": 1, "event": "table_file_creation", "file_number": 7, "file_size": 88718, "table_properties": {"data_size": 86833, "index_size": 362, "index_partitions": 0, "top_level_index_size": 0, "index_key_is_user_key": 0, "index_value_is_delta_encoded": 0, "filter_size": 709, "raw_key_size": 12384, "raw_average_key_size": 49, "raw_value_size": 79974, "raw_average_value_size": 321, "num_data_blocks": 11, "num_entries": 249, "num_deletions": 3, "num_merge_operands": 0, "num_range_deletions": 0, "format_version": 0, "fixed_key_len": 0, "filter_policy": "rocksdb.BuiltinBloomFilter", "column_family_name": "default", "column_family_id": 0, "comparator": "leveldb.BytewiseComparator", "merge_operator": "", "prefix_extractor_name": "nullptr", "property_collectors": "[]", "compression": "NoCompression", "compression_options": "window_bits=-14; level=32767; strategy=0; max_dict_bytes=0; zstd_max_train_bytes=0; enabled=0; ", "creation_time": 1706397866, "oldest_key_time": 3, "file_creation_time": 0}} 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.237+0000 7f852852b700 4 rocksdb: [version_set.cc:3826] Creating manifest 8 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.238+0000 7f852852b700 4 rocksdb: EVENT_LOG_v1 {"time_micros": 1706397866239530, "job": 1, "event": "recovery_finished"} 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.238+0000 7f852852b700 4 rocksdb: DB pointer 0x5572d6a0d800 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.239+0000 7f850f343700 4 rocksdb: [db_impl/db_impl.cc:850] ------- DUMPING STATS ------- 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.239+0000 7f850f343700 4 rocksdb: [db_impl/db_impl.cc:851] 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ** DB Stats ** 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Cumulative writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 GB, 0.00 MB/s 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Cumulative WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 GB, 0.00 MB/s 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Cumulative stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Interval writes: 0 writes, 0 keys, 0 commit groups, 0.0 writes per commit group, ingest: 0.00 MB, 0.00 MB/s 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Interval WAL: 0 writes, 0 syncs, 0.00 writes per sync, written: 0.00 MB, 0.00 MB/s 2024-01-27T23:24:26.604 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Interval stall: 00:00:0.000 H:M:S, 0.0 percent 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ** Compaction Stats [default] ** 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.5 0.00 0.00 1 0.001 0 0 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.5 0.00 0.00 1 0.001 0 0 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.5 0.00 0.00 1 0.001 0 0 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ** Compaction Stats [default] ** 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.5 0.00 0.00 1 0.001 0 0 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(Total Files): cumulative 0, interval 0 2024-01-27T23:24:26.605 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(Keys): cumulative 0, interval 0 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Cumulative compaction: 0.00 GB write, 30.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Interval compaction: 0.00 GB write, 30.28 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ** File Read Latency Histogram By Level [default] ** 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ** Compaction Stats [default] ** 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Level Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: L0 2/0 88.33 KB 0.5 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.5 0.00 0.00 1 0.001 0 0 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Sum 2/0 88.33 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 1.0 0.0 155.5 0.00 0.00 1 0.001 0 0 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Int 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.00 0.00 0 0.000 0 0 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ** Compaction Stats [default] ** 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Priority Files Size Score Read(GB) Rn(GB) Rnp1(GB) Write(GB) Wnew(GB) Moved(GB) W-Amp Rd(MB/s) Wr(MB/s) Comp(sec) CompMergeCPU(sec) Comp(cnt) Avg(sec) KeyIn KeyDrop 2024-01-27T23:24:26.606 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: User 0/0 0.00 KB 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 0.0 155.5 0.00 0.00 1 0.001 0 0 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Uptime(secs): 0.0 total, 0.0 interval 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Flush(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(GB): cumulative 0.000, interval 0.000 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(Total Files): cumulative 0, interval 0 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(L0 Files): cumulative 0, interval 0 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: AddFile(Keys): cumulative 0, interval 0 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Cumulative compaction: 0.00 GB write, 29.73 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Interval compaction: 0.00 GB write, 0.00 MB/s write, 0.00 GB read, 0.00 MB/s read, 0.0 seconds 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Stalls(count): 0 level0_slowdown, 0 level0_slowdown_with_compaction, 0 level0_numfiles, 0 level0_numfiles_with_compaction, 0 stop for pending_compaction_bytes, 0 slowdown for pending_compaction_bytes, 0 memtable_compaction, 0 memtable_slowdown, interval 0 total count 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ** File Read Latency Histogram By Level [default] ** 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.239+0000 7f852852b700 0 starting mon.smithi121 rank 0 at public addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] at bind addrs [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] mon_data /var/lib/ceph/mon/ceph-smithi121 fsid 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.240+0000 7f852852b700 1 mon.smithi121@-1(???) e1 preinit fsid 2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:26.607 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 systemd[1]: Started Ceph mon.smithi121 for 2005e414-bd6b-11ee-95b2-87774f69a715. 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.240+0000 7f852852b700 0 mon.smithi121@-1(???).mds e1 new map 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.240+0000 7f852852b700 0 mon.smithi121@-1(???).mds e1 print_map 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: e1 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: enable_multiple, ever_enabled_multiple: 1,1 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: compat: compat={},rocompat={},incompat={1=base v0.20,2=client writeable ranges,3=default file layouts on dirs,4=dir inode in separate object,5=mds uses versioned encoding,6=dirfrag is stored in omap,8=no anchor table,9=file layout v2,10=snaprealm v2} 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: legacy client fscid: -1 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: No filesystems configured 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.240+0000 7f852852b700 0 mon.smithi121@-1(???).osd e1 crush map has features 3314932999778484224, adjusting msgr requires 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.240+0000 7f852852b700 0 mon.smithi121@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.240+0000 7f852852b700 0 mon.smithi121@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.240+0000 7f852852b700 0 mon.smithi121@-1(???).osd e1 crush map has features 288514050185494528, adjusting msgr requires 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:24.293528+0000 mon.smithi121 (mon.0) 8 : audit [INF] from='client.? 172.21.15.121:0/2805705540' entity='client.admin' cmd=[{"prefix": "config assimilate-conf"}]: dispatch 2024-01-27T23:24:26.608 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:24.296717+0000 mon.smithi121 (mon.0) 9 : audit [INF] from='client.? 172.21.15.121:0/2805705540' entity='client.admin' cmd='[{"prefix": "config assimilate-conf"}]': finished 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.241+0000 7f852852b700 1 mon.smithi121@-1(???).paxosservice(auth 1..2) refresh upgraded, format 0 -> 3 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expand map: {default=false} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta from 'false' to 'false' 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expanded map: {default=false} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expand map: {default=info} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta from 'info' to 'info' 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expanded map: {default=info} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expand map: {default=daemon} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta from 'daemon' to 'daemon' 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expanded map: {default=daemon} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expand map: {default=debug} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta from 'debug' to 'debug' 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:24:26.242+0000 7f852852b700 20 expand_channel_meta expanded map: {default=debug} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:26.256796+0000 mon.smithi121 (mon.0) 1 : cluster [INF] mon.smithi121 is new leader, mons smithi121 in quorum (ranks 0) 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:26.256879+0000 mon.smithi121 (mon.0) 2 : cluster [DBG] monmap e1: 1 mons at {smithi121=[v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0]} 2024-01-27T23:24:26.609 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:26.256942+0000 mon.smithi121 (mon.0) 3 : cluster [DBG] fsmap 2024-01-27T23:24:26.610 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:26.256985+0000 mon.smithi121 (mon.0) 4 : cluster [DBG] osdmap e1: 0 total, 0 up, 0 in 2024-01-27T23:24:26.610 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:26.257429+0000 mon.smithi121 (mon.0) 5 : cluster [DBG] mgrmap e1: no daemons active 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.604+0000 7f1fc6c2a700 1 Processor -- start 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.605+0000 7f1fc6c2a700 1 -- start start 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.605+0000 7f1fc6c2a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fe7f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.605+0000 7f1fc6c2a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc0070790 con 0x7f1fc0070370 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.605+0000 7f1fc5c28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fe7f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.605+0000 7f1fc5c28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fe7f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42718/0 (socket says 172.21.15.121:42718) 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.605+0000 7f1fc5c28700 1 -- 172.21.15.121:0/1834749303 learned_addr learned my addr 172.21.15.121:0/1834749303 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.606+0000 7f1fc5c28700 1 -- 172.21.15.121:0/1834749303 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fc00fece0 con 0x7f1fc0070370 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.606+0000 7f1fc5c28700 1 --2- 172.21.15.121:0/1834749303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fe7f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1fbc00b5f0 tx=0x7f1fbc006ee0).ready entity=mon.0 client_cookie=637480255d41a7f4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.606+0000 7f1fc4c26700 1 -- 172.21.15.121:0/1834749303 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fbc00d600 con 0x7f1fc0070370 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.606+0000 7f1fc4c26700 1 -- 172.21.15.121:0/1834749303 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1fbc00dbd0 con 0x7f1fc0070370 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.606+0000 7f1fc4c26700 1 -- 172.21.15.121:0/1834749303 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fbc011c00 con 0x7f1fc0070370 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.607+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/1834749303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 msgr2=0x7f1fc00fe7f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.607+0000 7f1fc6c2a700 1 --2- 172.21.15.121:0/1834749303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fe7f0 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7f1fbc00b5f0 tx=0x7f1fbc006ee0).stop 2024-01-27T23:24:26.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.607+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/1834749303 shutdown_connections 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.607+0000 7f1fc6c2a700 1 --2- 172.21.15.121:0/1834749303 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fe7f0 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.607+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/1834749303 >> 172.21.15.121:0/1834749303 conn(0x7f1fc00f6210 msgr2=0x7f1fc00f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.607+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/1834749303 shutdown_connections 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.607+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/1834749303 wait complete. 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc6c2a700 1 Processor -- start 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc6c2a700 1 -- start start 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc6c2a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fc270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc6c2a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1fc00fa6f0 con 0x7f1fc0070370 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc5c28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fc270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc5c28700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fc270 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42732/0 (socket says 172.21.15.121:42732) 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc5c28700 1 -- 172.21.15.121:0/3140482311 learned_addr learned my addr 172.21.15.121:0/3140482311 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc5c28700 1 -- 172.21.15.121:0/3140482311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1fbc00b200 con 0x7f1fc0070370 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.608+0000 7f1fc5c28700 1 --2- 172.21.15.121:0/3140482311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fc270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1fbc00e040 tx=0x7f1fbc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.609+0000 7f1fae7fc700 1 -- 172.21.15.121:0/3140482311 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fbc00d9f0 con 0x7f1fc0070370 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.609+0000 7f1fae7fc700 1 -- 172.21.15.121:0/3140482311 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f1fbc01a9f0 con 0x7f1fc0070370 2024-01-27T23:24:26.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.609+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1fc00fa890 con 0x7f1fc0070370 2024-01-27T23:24:26.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.609+0000 7f1fae7fc700 1 -- 172.21.15.121:0/3140482311 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1fbc01b890 con 0x7f1fc0070370 2024-01-27T23:24:26.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.609+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1fc00faca0 con 0x7f1fc0070370 2024-01-27T23:24:26.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.609+0000 7f1fae7fc700 1 -- 172.21.15.121:0/3140482311 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f1fbc00f070 con 0x7f1fc0070370 2024-01-27T23:24:26.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.609+0000 7f1fae7fc700 1 -- 172.21.15.121:0/3140482311 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f1fbc025b30 con 0x7f1fc0070370 2024-01-27T23:24:26.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.610+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1fa0004fa0 con 0x7f1fc0070370 2024-01-27T23:24:26.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.612+0000 7f1fae7fc700 1 -- 172.21.15.121:0/3140482311 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f1fbc016070 con 0x7f1fc0070370 2024-01-27T23:24:26.665 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.676+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=public_network}] v 0) v1 -- 0x7f1fa0005bf0 con 0x7f1fc0070370 2024-01-27T23:24:26.670 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.681+0000 7f1fae7fc700 1 -- 172.21.15.121:0/3140482311 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=public_network}]=0 v3)=0 v3) v1 ==== 124+0+0 (secure 0 0 0) 0x7f1fbc025cf0 con 0x7f1fc0070370 2024-01-27T23:24:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.682+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 msgr2=0x7f1fc00fc270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.682+0000 7f1fc6c2a700 1 --2- 172.21.15.121:0/3140482311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fc270 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f1fbc00e040 tx=0x7f1fbc006ee0).stop 2024-01-27T23:24:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.682+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 shutdown_connections 2024-01-27T23:24:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.682+0000 7f1fc6c2a700 1 --2- 172.21.15.121:0/3140482311 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1fc0070370 0x7f1fc00fc270 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:26.671 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.682+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 >> 172.21.15.121:0/3140482311 conn(0x7f1fc00f6210 msgr2=0x7f1fc00f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.682+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 shutdown_connections 2024-01-27T23:24:26.672 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:26.682+0000 7f1fc6c2a700 1 -- 172.21.15.121:0/3140482311 wait complete. 2024-01-27T23:24:27.030 INFO:teuthology.orchestra.run.smithi121.stderr:Wrote config to /etc/ceph/ceph.conf 2024-01-27T23:24:27.030 INFO:teuthology.orchestra.run.smithi121.stderr:Wrote keyring to /etc/ceph/ceph.client.admin.keyring 2024-01-27T23:24:27.030 INFO:teuthology.orchestra.run.smithi121.stderr:Creating mgr... 2024-01-27T23:24:27.030 INFO:teuthology.orchestra.run.smithi121.stderr:Verifying port 9283 ... 2024-01-27T23:24:27.032 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl daemon-reload 2024-01-27T23:24:27.125 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mgr.smithi121.hcppvz 2024-01-27T23:24:27.148 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl reset-failed ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mgr.smithi121.hcppvz 2024-01-27T23:24:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Failed to reset failed state of unit ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mgr.smithi121.hcppvz.service: Unit ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mgr.smithi121.hcppvz.service not loaded. 2024-01-27T23:24:27.154 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl enable ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mgr.smithi121.hcppvz 2024-01-27T23:24:27.160 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: Created symlink /etc/systemd/system/ceph-2005e414-bd6b-11ee-95b2-87774f69a715.target.wants/ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mgr.smithi121.hcppvz.service → /etc/systemd/system/ceph-2005e414-bd6b-11ee-95b2-87774f69a715@.service. 2024-01-27T23:24:27.243 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl start ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mgr.smithi121.hcppvz 2024-01-27T23:24:27.589 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-27T23:24:27.597 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: disabled 2024-01-27T23:24:27.598 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active firewalld.service 2024-01-27T23:24:27.605 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-01-27T23:24:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld.service is not enabled 2024-01-27T23:24:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to enable service . firewalld.service is not available 2024-01-27T23:24:27.606 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to open ports <[]>. firewalld.service is not available 2024-01-27T23:24:27.607 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-27T23:24:27.614 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: disabled 2024-01-27T23:24:27.615 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active firewalld.service 2024-01-27T23:24:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-01-27T23:24:27.623 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld.service is not enabled 2024-01-27T23:24:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to open ports <[9283]>. firewalld.service is not available 2024-01-27T23:24:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr to start... 2024-01-27T23:24:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr... 2024-01-27T23:24:27.624 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-27T23:24:27.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:26.682251+0000 mon.smithi121 (mon.0) 6 : audit [INF] from='client.? 172.21.15.121:0/3140482311' entity='client.admin' 2024-01-27T23:24:28.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.020+0000 7f583952b700 1 Processor -- start 2024-01-27T23:24:28.011 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.020+0000 7f583952b700 1 -- start start 2024-01-27T23:24:28.011 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.020+0000 7f583952b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f5834100b90 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:28.011 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.020+0000 7f583952b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f58340fc580 con 0x7f58340703d0 2024-01-27T23:24:28.012 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.020+0000 7f5833fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f5834100b90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:28.012 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.021+0000 7f5833fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f5834100b90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42750/0 (socket says 172.21.15.121:42750) 2024-01-27T23:24:28.012 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.021+0000 7f5833fff700 1 -- 172.21.15.121:0/1403961305 learned_addr learned my addr 172.21.15.121:0/1403961305 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:28.012 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.021+0000 7f5833fff700 1 -- 172.21.15.121:0/1403961305 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5834101080 con 0x7f58340703d0 2024-01-27T23:24:28.012 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.021+0000 7f5833fff700 1 --2- 172.21.15.121:0/1403961305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f5834100b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f582400b5f0 tx=0x7f5824006ee0).ready entity=mon.0 client_cookie=4a0ca9a262788307 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:28.012 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.021+0000 7f5832ffd700 1 -- 172.21.15.121:0/1403961305 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582400d600 con 0x7f58340703d0 2024-01-27T23:24:28.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.021+0000 7f5832ffd700 1 -- 172.21.15.121:0/1403961305 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f582400dbd0 con 0x7f58340703d0 2024-01-27T23:24:28.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.021+0000 7f5832ffd700 1 -- 172.21.15.121:0/1403961305 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5824011c00 con 0x7f58340703d0 2024-01-27T23:24:28.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 -- 172.21.15.121:0/1403961305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 msgr2=0x7f5834100b90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:28.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 --2- 172.21.15.121:0/1403961305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f5834100b90 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7f582400b5f0 tx=0x7f5824006ee0).stop 2024-01-27T23:24:28.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 -- 172.21.15.121:0/1403961305 shutdown_connections 2024-01-27T23:24:28.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 --2- 172.21.15.121:0/1403961305 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f5834100b90 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:28.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 -- 172.21.15.121:0/1403961305 >> 172.21.15.121:0/1403961305 conn(0x7f58340f6210 msgr2=0x7f58340f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:28.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 -- 172.21.15.121:0/1403961305 shutdown_connections 2024-01-27T23:24:28.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 -- 172.21.15.121:0/1403961305 wait complete. 2024-01-27T23:24:28.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 Processor -- start 2024-01-27T23:24:28.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.022+0000 7f583952b700 1 -- start start 2024-01-27T23:24:28.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f583952b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f583410ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f583952b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f583410cf60 con 0x7f58340703d0 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5833fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f583410ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5833fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f583410ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42760/0 (socket says 172.21.15.121:42760) 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5833fff700 1 -- 172.21.15.121:0/2510544247 learned_addr learned my addr 172.21.15.121:0/2510544247 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5833fff700 1 -- 172.21.15.121:0/2510544247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f582400b200 con 0x7f58340703d0 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5833fff700 1 --2- 172.21.15.121:0/2510544247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f583410ca70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f582400b190 tx=0x7f5824006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5830ff9700 1 -- 172.21.15.121:0/2510544247 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582401b410 con 0x7f58340703d0 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f583410d160 con 0x7f58340703d0 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5830ff9700 1 -- 172.21.15.121:0/2510544247 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f582401b9e0 con 0x7f58340703d0 2024-01-27T23:24:28.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f5830ff9700 1 -- 172.21.15.121:0/2510544247 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f582401a880 con 0x7f58340703d0 2024-01-27T23:24:28.016 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.023+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f583410d570 con 0x7f58340703d0 2024-01-27T23:24:28.016 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.024+0000 7f5830ff9700 1 -- 172.21.15.121:0/2510544247 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7f582400f070 con 0x7f58340703d0 2024-01-27T23:24:28.016 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.024+0000 7f5830ff9700 1 -- 172.21.15.121:0/2510544247 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f5824025c60 con 0x7f58340703d0 2024-01-27T23:24:28.016 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.024+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5834045ad0 con 0x7f58340703d0 2024-01-27T23:24:28.016 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.026+0000 7f5830ff9700 1 -- 172.21.15.121:0/2510544247 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7f5824016070 con 0x7f58340703d0 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.095+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7f5834106840 con 0x7f58340703d0 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.095+0000 7f5830ff9700 1 -- 172.21.15.121:0/2510544247 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7f582401bb50 con 0x7f58340703d0 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsid": "2005e414-bd6b-11ee-95b2-87774f69a715", 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "health": { 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "checks": {}, 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mutes": [] 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:28.091 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum": [ 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 0 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "smithi121" 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_age": 1, 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "monmap": { 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osdmap": { 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:28.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgmap": { 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-27T23:24:28.093 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsmap": { 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": false, 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modules": [ 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "iostat", 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "restful" 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:28.094 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "servicemap": { 2024-01-27T23:24:28.095 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:28.095 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modified": "2024-01-27T23:24:23.059075+0000", 2024-01-27T23:24:28.095 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.096+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 msgr2=0x7f583410ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.096+0000 7f583952b700 1 --2- 172.21.15.121:0/2510544247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f583410ca70 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f582400b190 tx=0x7f5824006ee0).stop 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.096+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 shutdown_connections 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.096+0000 7f583952b700 1 --2- 172.21.15.121:0/2510544247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f58340703d0 0x7f583410ca70 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.096+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 >> 172.21.15.121:0/2510544247 conn(0x7f58340f6210 msgr2=0x7f58340f7ca0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.096+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 shutdown_connections 2024-01-27T23:24:28.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:28.096+0000 7f583952b700 1 -- 172.21.15.121:0/2510544247 wait complete. 2024-01-27T23:24:28.472 INFO:teuthology.orchestra.run.smithi121.stderr:mgr not available, waiting (1/15)... 2024-01-27T23:24:29.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:28.096715+0000 mon.smithi121 (mon.0) 7 : audit [DBG] from='client.? 172.21.15.121:0/2510544247' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-27T23:24:30.475 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-27T23:24:30.869 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.879+0000 7feb25679700 1 Processor -- start 2024-01-27T23:24:30.869 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.879+0000 7feb25679700 1 -- start start 2024-01-27T23:24:30.869 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.879+0000 7feb25679700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb200fadf0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:30.869 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.879+0000 7feb25679700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb200fb2e0 con 0x7feb200fa9d0 2024-01-27T23:24:30.869 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.879+0000 7feb1ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb200fadf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:30.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.879+0000 7feb1ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb200fadf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42770/0 (socket says 172.21.15.121:42770) 2024-01-27T23:24:30.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.879+0000 7feb1ffff700 1 -- 172.21.15.121:0/1839791455 learned_addr learned my addr 172.21.15.121:0/1839791455 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:30.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.880+0000 7feb1ffff700 1 -- 172.21.15.121:0/1839791455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb200fb420 con 0x7feb200fa9d0 2024-01-27T23:24:30.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.880+0000 7feb1ffff700 1 --2- 172.21.15.121:0/1839791455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb200fadf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7feb1000b5f0 tx=0x7feb10006ee0).ready entity=mon.0 client_cookie=d33caf339d35688d server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:30.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.880+0000 7feb1effd700 1 -- 172.21.15.121:0/1839791455 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb1000d600 con 0x7feb200fa9d0 2024-01-27T23:24:30.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.880+0000 7feb1effd700 1 -- 172.21.15.121:0/1839791455 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb1000dbd0 con 0x7feb200fa9d0 2024-01-27T23:24:30.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.880+0000 7feb1effd700 1 -- 172.21.15.121:0/1839791455 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb10011c00 con 0x7feb200fa9d0 2024-01-27T23:24:30.871 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.881+0000 7feb25679700 1 -- 172.21.15.121:0/1839791455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 msgr2=0x7feb200fadf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.881+0000 7feb25679700 1 --2- 172.21.15.121:0/1839791455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb200fadf0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7feb1000b5f0 tx=0x7feb10006ee0).stop 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 -- 172.21.15.121:0/1839791455 shutdown_connections 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 --2- 172.21.15.121:0/1839791455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb200fadf0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 -- 172.21.15.121:0/1839791455 >> 172.21.15.121:0/1839791455 conn(0x7feb200f6210 msgr2=0x7feb200f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 -- 172.21.15.121:0/1839791455 shutdown_connections 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 -- 172.21.15.121:0/1839791455 wait complete. 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 Processor -- start 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 -- start start 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb2010ca70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.882+0000 7feb25679700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb2010cf60 con 0x7feb200fa9d0 2024-01-27T23:24:30.872 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb2010ca70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb2010ca70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:42774/0 (socket says 172.21.15.121:42774) 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1ffff700 1 -- 172.21.15.121:0/2944927346 learned_addr learned my addr 172.21.15.121:0/2944927346 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1ffff700 1 -- 172.21.15.121:0/2944927346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb1000b200 con 0x7feb200fa9d0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1ffff700 1 --2- 172.21.15.121:0/2944927346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb2010ca70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feb1000e040 tx=0x7feb10006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1cff9700 1 -- 172.21.15.121:0/2944927346 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb1001b410 con 0x7feb200fa9d0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb2010d160 con 0x7feb200fa9d0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1cff9700 1 -- 172.21.15.121:0/2944927346 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7feb1001b9e0 con 0x7feb200fa9d0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1cff9700 1 -- 172.21.15.121:0/2944927346 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb1001a820 con 0x7feb200fa9d0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb2010d570 con 0x7feb200fa9d0 2024-01-27T23:24:30.873 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1cff9700 1 -- 172.21.15.121:0/2944927346 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 1) v1 ==== 484+0+0 (secure 0 0 0) 0x7feb1000f070 con 0x7feb200fa9d0 2024-01-27T23:24:30.874 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.883+0000 7feb1cff9700 1 -- 172.21.15.121:0/2944927346 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7feb10025bd0 con 0x7feb200fa9d0 2024-01-27T23:24:30.874 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.884+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb20045ad0 con 0x7feb200fa9d0 2024-01-27T23:24:30.875 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.886+0000 7feb1cff9700 1 -- 172.21.15.121:0/2944927346 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+72097 (secure 0 0 0) 0x7feb10016070 con 0x7feb200fa9d0 2024-01-27T23:24:30.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.955+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7feb20106b80 con 0x7feb200fa9d0 2024-01-27T23:24:30.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.956+0000 7feb1cff9700 1 -- 172.21.15.121:0/2944927346 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1230 (secure 0 0 0) 0x7feb10025e60 con 0x7feb200fa9d0 2024-01-27T23:24:30.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-01-27T23:24:30.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsid": "2005e414-bd6b-11ee-95b2-87774f69a715", 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "health": { 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "checks": {}, 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mutes": [] 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum": [ 2024-01-27T23:24:30.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 0 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "smithi121" 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_age": 4, 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "monmap": { 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osdmap": { 2024-01-27T23:24:30.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:30.949 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-27T23:24:30.949 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-27T23:24:30.949 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgmap": { 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-27T23:24:30.953 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsmap": { 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": false, 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modules": [ 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "iostat", 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "restful" 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "servicemap": { 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:30.954 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modified": "2024-01-27T23:24:23.059075+0000", 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.957+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 msgr2=0x7feb2010ca70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.957+0000 7feb25679700 1 --2- 172.21.15.121:0/2944927346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb2010ca70 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7feb1000e040 tx=0x7feb10006ee0).stop 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.957+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 shutdown_connections 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.957+0000 7feb25679700 1 --2- 172.21.15.121:0/2944927346 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb200fa9d0 0x7feb2010ca70 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.957+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 >> 172.21.15.121:0/2944927346 conn(0x7feb200f6210 msgr2=0x7feb20070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.957+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 shutdown_connections 2024-01-27T23:24:30.955 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:30.957+0000 7feb25679700 1 -- 172.21.15.121:0/2944927346 wait complete. 2024-01-27T23:24:31.289 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:31 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:30.956156+0000 mon.smithi121 (mon.0) 8 : audit [DBG] from='client.? 172.21.15.121:0/2944927346' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-27T23:24:31.320 INFO:teuthology.orchestra.run.smithi121.stderr:mgr not available, waiting (2/15)... 2024-01-27T23:24:32.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:32.534042+0000 mon.smithi121 (mon.0) 9 : cluster [INF] Activating manager daemon smithi121.hcppvz 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:32.536567+0000 mon.smithi121 (mon.0) 10 : cluster [DBG] mgrmap e2: smithi121.hcppvz(active, starting, since 0.00263957s) 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.537547+0000 mon.smithi121 (mon.0) 11 : audit [DBG] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.537681+0000 mon.smithi121 (mon.0) 12 : audit [DBG] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.537796+0000 mon.smithi121 (mon.0) 13 : audit [DBG] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.537905+0000 mon.smithi121 (mon.0) 14 : audit [DBG] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.538190+0000 mon.smithi121 (mon.0) 15 : audit [DBG] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mgr metadata", "who": "smithi121.hcppvz", "id": "smithi121.hcppvz"}]: dispatch 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:32.551548+0000 mon.smithi121 (mon.0) 16 : cluster [INF] Manager daemon smithi121.hcppvz is now available 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.557962+0000 mon.smithi121 (mon.0) 17 : audit [INF] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.562928+0000 mon.smithi121 (mon.0) 18 : audit [INF] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.564013+0000 mon.smithi121 (mon.0) 19 : audit [INF] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.568689+0000 mon.smithi121 (mon.0) 20 : audit [INF] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:32.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:32.570664+0000 mon.smithi121 (mon.0) 21 : audit [INF] from='mgr.14100 172.21.15.121:0/2690603556' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:33.323 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 status -f json-pretty 2024-01-27T23:24:34.773 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.782+0000 7efc59d6f700 1 Processor -- start 2024-01-27T23:24:34.773 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.782+0000 7efc59d6f700 1 -- start start 2024-01-27T23:24:34.773 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.782+0000 7efc59d6f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc54070a70 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:34.773 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc59d6f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc540728f0 con 0x7efc540724d0 2024-01-27T23:24:34.773 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc54070a70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc54070a70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53516/0 (socket says 172.21.15.121:53516) 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc537fe700 1 -- 172.21.15.121:0/2235300602 learned_addr learned my addr 172.21.15.121:0/2235300602 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc537fe700 1 -- 172.21.15.121:0/2235300602 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc54070f60 con 0x7efc540724d0 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc537fe700 1 --2- 172.21.15.121:0/2235300602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc54070a70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efc3c00b5f0 tx=0x7efc3c006ee0).ready entity=mon.0 client_cookie=24f01009859a286f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc527fc700 1 -- 172.21.15.121:0/2235300602 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc3c00d600 con 0x7efc540724d0 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc527fc700 1 -- 172.21.15.121:0/2235300602 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc3c00dbd0 con 0x7efc540724d0 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.783+0000 7efc527fc700 1 -- 172.21.15.121:0/2235300602 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc3c011c00 con 0x7efc540724d0 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.784+0000 7efc59d6f700 1 -- 172.21.15.121:0/2235300602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 msgr2=0x7efc54070a70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.784+0000 7efc59d6f700 1 --2- 172.21.15.121:0/2235300602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc54070a70 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7efc3c00b5f0 tx=0x7efc3c006ee0).stop 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.784+0000 7efc59d6f700 1 -- 172.21.15.121:0/2235300602 shutdown_connections 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.784+0000 7efc59d6f700 1 --2- 172.21.15.121:0/2235300602 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc54070a70 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.784+0000 7efc59d6f700 1 -- 172.21.15.121:0/2235300602 >> 172.21.15.121:0/2235300602 conn(0x7efc540f74c0 msgr2=0x7efc540f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.784+0000 7efc59d6f700 1 -- 172.21.15.121:0/2235300602 shutdown_connections 2024-01-27T23:24:34.774 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.784+0000 7efc59d6f700 1 -- 172.21.15.121:0/2235300602 wait complete. 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.785+0000 7efc59d6f700 1 Processor -- start 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.785+0000 7efc59d6f700 1 -- start start 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.785+0000 7efc59d6f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc5410cd30 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.785+0000 7efc59d6f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efc5410d220 con 0x7efc540724d0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.785+0000 7efc537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc5410cd30 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.785+0000 7efc537fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc5410cd30 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53524/0 (socket says 172.21.15.121:53524) 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.785+0000 7efc537fe700 1 -- 172.21.15.121:0/2264644897 learned_addr learned my addr 172.21.15.121:0/2264644897 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.786+0000 7efc537fe700 1 -- 172.21.15.121:0/2264644897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efc3c00b200 con 0x7efc540724d0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.786+0000 7efc537fe700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc5410cd30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7efc3c006c70 tx=0x7efc3c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.786+0000 7efc4bfff700 1 -- 172.21.15.121:0/2264644897 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc3c00da30 con 0x7efc540724d0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.786+0000 7efc4bfff700 1 -- 172.21.15.121:0/2264644897 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7efc3c011690 con 0x7efc540724d0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.786+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efc5410d420 con 0x7efc540724d0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.786+0000 7efc4bfff700 1 -- 172.21.15.121:0/2264644897 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efc3c01b7b0 con 0x7efc540724d0 2024-01-27T23:24:34.777 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.786+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efc5410d830 con 0x7efc540724d0 2024-01-27T23:24:34.778 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.787+0000 7efc4bfff700 1 -- 172.21.15.121:0/2264644897 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 3) v1 ==== 36029+0+0 (secure 0 0 0) 0x7efc3c00f070 con 0x7efc540724d0 2024-01-27T23:24:34.778 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.787+0000 7efc4bfff700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7efc3402fcc0 0x7efc34032180 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:34.778 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.787+0000 7efc4bfff700 1 -- 172.21.15.121:0/2264644897 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7efc3c040f90 con 0x7efc540724d0 2024-01-27T23:24:34.778 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.787+0000 7efc52ffd700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7efc3402fcc0 0x7efc34032180 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:34.778 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.787+0000 7efc52ffd700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7efc3402fcc0 0x7efc34032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efc440096a0 tx=0x7efc44006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:34.778 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.787+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efc54045ad0 con 0x7efc540724d0 2024-01-27T23:24:34.780 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.792+0000 7efc4bfff700 1 -- 172.21.15.121:0/2264644897 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efc3c016070 con 0x7efc540724d0 2024-01-27T23:24:34.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:33.539580+0000 mon.smithi121 (mon.0) 22 : cluster [DBG] mgrmap e3: smithi121.hcppvz(active, since 1.00565s) 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.994+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "status", "format": "json-pretty"} v 0) v1 -- 0x7efc54101e00 con 0x7efc540724d0 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.995+0000 7efc4bfff700 1 -- 172.21.15.121:0/2264644897 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "status", "format": "json-pretty"}]=0 v0) v1 ==== 79+0+1229 (secure 0 0 0) 0x7efc54101e00 con 0x7efc540724d0 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsid": "2005e414-bd6b-11ee-95b2-87774f69a715", 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "health": { 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "status": "HEALTH_OK", 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "checks": {}, 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mutes": [] 2024-01-27T23:24:34.990 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "election_epoch": 5, 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum": [ 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 0 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_names": [ 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "smithi121" 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "quorum_age": 8, 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "monmap": { 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "min_mon_release_name": "pacific", 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_mons": 1 2024-01-27T23:24:34.991 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osdmap": { 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_osds": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_up_osds": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_up_since": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_in_osds": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "osd_in_since": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_remapped_pgs": 0 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgmap": { 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "pgs_by_state": [], 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pgs": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_pools": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_objects": 0, 2024-01-27T23:24:34.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "data_bytes": 0, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_used": 0, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_avail": 0, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "bytes_total": 0 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "fsmap": { 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "by_rank": [], 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "up:standby": 0 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap": { 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": true, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standbys": 0, 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modules": [ 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "iostat", 2024-01-27T23:24:34.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "restful" 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ], 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "servicemap": { 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 1, 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "modified": "2024-01-27T23:24:23.059075+0000", 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "services": {} 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: }, 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "progress_events": {} 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.996+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7efc3402fcc0 msgr2=0x7efc34032180 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.996+0000 7efc59d6f700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7efc3402fcc0 0x7efc34032180 secure :-1 s=READY pgs=5 cs=0 l=1 rev1=1 rx=0x7efc440096a0 tx=0x7efc44006b40).stop 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.996+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 msgr2=0x7efc5410cd30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:34.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.996+0000 7efc59d6f700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc5410cd30 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7efc3c006c70 tx=0x7efc3c006ee0).stop 2024-01-27T23:24:34.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.997+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 shutdown_connections 2024-01-27T23:24:34.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.997+0000 7efc59d6f700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7efc3402fcc0 0x7efc34032180 unknown :-1 s=CLOSED pgs=5 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:34.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.997+0000 7efc59d6f700 1 --2- 172.21.15.121:0/2264644897 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efc540724d0 0x7efc5410cd30 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:34.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.997+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 >> 172.21.15.121:0/2264644897 conn(0x7efc540f74c0 msgr2=0x7efc540f9060 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:34.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.997+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 shutdown_connections 2024-01-27T23:24:34.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:34.997+0000 7efc59d6f700 1 -- 172.21.15.121:0/2264644897 wait complete. 2024-01-27T23:24:35.353 INFO:teuthology.orchestra.run.smithi121.stderr:mgr is available 2024-01-27T23:24:35.353 INFO:teuthology.orchestra.run.smithi121.stderr:Enabling cephadm module... 2024-01-27T23:24:35.353 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable cephadm 2024-01-27T23:24:35.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:35 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:34.995348+0000 mon.smithi121 (mon.0) 23 : audit [DBG] from='client.? 172.21.15.121:0/2264644897' entity='client.admin' cmd=[{"prefix": "status", "format": "json-pretty"}]: dispatch 2024-01-27T23:24:35.702 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.710+0000 7faf4f59e700 1 Processor -- start 2024-01-27T23:24:35.702 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.710+0000 7faf4f59e700 1 -- start start 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4f59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf500fe730 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4f59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf500fec20 con 0x7faf50070430 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf500fe730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf500fe730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53532/0 (socket says 172.21.15.121:53532) 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4e59c700 1 -- 172.21.15.121:0/3362748105 learned_addr learned my addr 172.21.15.121:0/3362748105 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4e59c700 1 -- 172.21.15.121:0/3362748105 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf500fed60 con 0x7faf50070430 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4e59c700 1 --2- 172.21.15.121:0/3362748105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf500fe730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7faf3800b5f0 tx=0x7faf38006ee0).ready entity=mon.0 client_cookie=7d883b1fffa44042 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4d59a700 1 -- 172.21.15.121:0/3362748105 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf3800d600 con 0x7faf50070430 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4d59a700 1 -- 172.21.15.121:0/3362748105 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faf3800dbd0 con 0x7faf50070430 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.711+0000 7faf4d59a700 1 -- 172.21.15.121:0/3362748105 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf38011c00 con 0x7faf50070430 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.712+0000 7faf4f59e700 1 -- 172.21.15.121:0/3362748105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 msgr2=0x7faf500fe730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.712+0000 7faf4f59e700 1 --2- 172.21.15.121:0/3362748105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf500fe730 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7faf3800b5f0 tx=0x7faf38006ee0).stop 2024-01-27T23:24:35.703 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.712+0000 7faf4f59e700 1 -- 172.21.15.121:0/3362748105 shutdown_connections 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.712+0000 7faf4f59e700 1 --2- 172.21.15.121:0/3362748105 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf500fe730 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.712+0000 7faf4f59e700 1 -- 172.21.15.121:0/3362748105 >> 172.21.15.121:0/3362748105 conn(0x7faf500f6210 msgr2=0x7faf500f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.712+0000 7faf4f59e700 1 -- 172.21.15.121:0/3362748105 shutdown_connections 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.712+0000 7faf4f59e700 1 -- 172.21.15.121:0/3362748105 wait complete. 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4f59e700 1 Processor -- start 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4f59e700 1 -- start start 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4f59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf5010ca90 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4f59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7faf5010cf80 con 0x7faf50070430 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf5010ca90 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf5010ca90 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53548/0 (socket says 172.21.15.121:53548) 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4e59c700 1 -- 172.21.15.121:0/1633779619 learned_addr learned my addr 172.21.15.121:0/1633779619 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4e59c700 1 -- 172.21.15.121:0/1633779619 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7faf3800b200 con 0x7faf50070430 2024-01-27T23:24:35.704 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf4e59c700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf5010ca90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7faf3800e040 tx=0x7faf38006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:35.705 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.713+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf3800d9f0 con 0x7faf50070430 2024-01-27T23:24:35.705 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.714+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7faf3801a9f0 con 0x7faf50070430 2024-01-27T23:24:35.705 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.714+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7faf5010d180 con 0x7faf50070430 2024-01-27T23:24:35.705 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.714+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7faf3801b8c0 con 0x7faf50070430 2024-01-27T23:24:35.705 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.714+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7faf5010d590 con 0x7faf50070430 2024-01-27T23:24:35.706 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.714+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7faf50045ad0 con 0x7faf50070430 2024-01-27T23:24:35.706 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.715+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 4) v1 ==== 36109+0+0 (secure 0 0 0) 0x7faf3800f070 con 0x7faf50070430 2024-01-27T23:24:35.706 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.715+0000 7faf46ffd700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7faf3002fd50 0x7faf30032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:35.706 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.715+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7faf38041960 con 0x7faf50070430 2024-01-27T23:24:35.706 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.715+0000 7faf4dd9b700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7faf3002fd50 0x7faf30032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:35.706 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.715+0000 7faf4dd9b700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7faf3002fd50 0x7faf30032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faf400096a0 tx=0x7faf40006b40).ready entity=mgr.14100 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:35.707 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.719+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7faf3802dd90 con 0x7faf50070430 2024-01-27T23:24:35.894 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:35.905+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "cephadm"} v 0) v1 -- 0x7faf50106a40 con 0x7faf50070430 2024-01-27T23:24:36.530 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.541+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7faf3801ab60 con 0x7faf50070430 2024-01-27T23:24:36.530 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.542+0000 7faf46ffd700 1 -- 172.21.15.121:0/1633779619 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "cephadm"}]=0 v5) v1 ==== 86+0+0 (secure 0 0 0) 0x7faf3802f840 con 0x7faf50070430 2024-01-27T23:24:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7faf3002fd50 msgr2=0x7faf30032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7faf3002fd50 0x7faf30032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7faf400096a0 tx=0x7faf40006b40).stop 2024-01-27T23:24:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 msgr2=0x7faf5010ca90 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf5010ca90 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7faf3800e040 tx=0x7faf38006ee0).stop 2024-01-27T23:24:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 shutdown_connections 2024-01-27T23:24:36.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7faf3002fd50 0x7faf30032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 --2- 172.21.15.121:0/1633779619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7faf50070430 0x7faf5010ca90 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 >> 172.21.15.121:0/1633779619 conn(0x7faf500f6210 msgr2=0x7faf500f7d20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 shutdown_connections 2024-01-27T23:24:36.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:36.544+0000 7faf4f59e700 1 -- 172.21.15.121:0/1633779619 wait complete. 2024-01-27T23:24:36.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:35.540793+0000 mon.smithi121 (mon.0) 24 : cluster [DBG] mgrmap e4: smithi121.hcppvz(active, since 3s) 2024-01-27T23:24:36.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:35.905703+0000 mon.smithi121 (mon.0) 25 : audit [INF] from='client.? 172.21.15.121:0/1633779619' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "cephadm"}]: dispatch 2024-01-27T23:24:36.919 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-27T23:24:37.328 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.338+0000 7fa60a253700 1 Processor -- start 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.338+0000 7fa60a253700 1 -- start start 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.339+0000 7fa60a253700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa604100d60 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.339+0000 7fa60a253700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa6040fc750 con 0x7fa6040703b0 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.339+0000 7fa6037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa604100d60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.339+0000 7fa6037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa604100d60 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53570/0 (socket says 172.21.15.121:53570) 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.339+0000 7fa6037fe700 1 -- 172.21.15.121:0/2025129495 learned_addr learned my addr 172.21.15.121:0/2025129495 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.339+0000 7fa6037fe700 1 -- 172.21.15.121:0/2025129495 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa604101250 con 0x7fa6040703b0 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.339+0000 7fa6037fe700 1 --2- 172.21.15.121:0/2025129495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa604100d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa5f400b5f0 tx=0x7fa5f4006ee0).ready entity=mon.0 client_cookie=f2c230fb8b36a658 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.340+0000 7fa6027fc700 1 -- 172.21.15.121:0/2025129495 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f400d600 con 0x7fa6040703b0 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.340+0000 7fa6027fc700 1 -- 172.21.15.121:0/2025129495 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa5f400dbd0 con 0x7fa6040703b0 2024-01-27T23:24:37.329 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.340+0000 7fa6027fc700 1 -- 172.21.15.121:0/2025129495 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f4011c00 con 0x7fa6040703b0 2024-01-27T23:24:37.330 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.340+0000 7fa60a253700 1 -- 172.21.15.121:0/2025129495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 msgr2=0x7fa604100d60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.340+0000 7fa60a253700 1 --2- 172.21.15.121:0/2025129495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa604100d60 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7fa5f400b5f0 tx=0x7fa5f4006ee0).stop 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 -- 172.21.15.121:0/2025129495 shutdown_connections 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 --2- 172.21.15.121:0/2025129495 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa604100d60 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 -- 172.21.15.121:0/2025129495 >> 172.21.15.121:0/2025129495 conn(0x7fa6040f63c0 msgr2=0x7fa6040f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 -- 172.21.15.121:0/2025129495 shutdown_connections 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 -- 172.21.15.121:0/2025129495 wait complete. 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 Processor -- start 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 -- start start 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa60410cc10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.341+0000 7fa60a253700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa60410d100 con 0x7fa6040703b0 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa6037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa60410cc10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa6037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa60410cc10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53578/0 (socket says 172.21.15.121:53578) 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa6037fe700 1 -- 172.21.15.121:0/48335562 learned_addr learned my addr 172.21.15.121:0/48335562 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:37.331 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa6037fe700 1 -- 172.21.15.121:0/48335562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa5f400b200 con 0x7fa6040703b0 2024-01-27T23:24:37.332 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa6037fe700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa60410cc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa5f400b190 tx=0x7fa5f4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa5ebfff700 1 -- 172.21.15.121:0/48335562 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f401b410 con 0x7fa6040703b0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa60410d300 con 0x7fa6040703b0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa5ebfff700 1 -- 172.21.15.121:0/48335562 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa5f401b9e0 con 0x7fa6040703b0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa5ebfff700 1 -- 172.21.15.121:0/48335562 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa5f401a880 con 0x7fa6040703b0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.342+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa60410d710 con 0x7fa6040703b0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.343+0000 7fa5ebfff700 1 -- 172.21.15.121:0/48335562 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa5f400f070 con 0x7fa6040703b0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.343+0000 7fa5ebfff700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 0x7fa5f0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.343+0000 7fa5ebfff700 1 -- 172.21.15.121:0/48335562 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7fa5f4041230 con 0x7fa6040703b0 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.343+0000 7fa602ffd700 1 -- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 msgr2=0x7fa5f0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3662234193 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.343+0000 7fa602ffd700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 0x7fa5f0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-27T23:24:37.333 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.343+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa604045ad0 con 0x7fa6040703b0 2024-01-27T23:24:37.340 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.351+0000 7fa5ebfff700 1 -- 172.21.15.121:0/48335562 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa5f4016070 con 0x7fa6040703b0 2024-01-27T23:24:37.532 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.544+0000 7fa602ffd700 1 -- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 msgr2=0x7fa5f0032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3662234193 2024-01-27T23:24:37.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.544+0000 7fa602ffd700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 0x7fa5f0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-27T23:24:37.533 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.545+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fa604106a00 con 0x7fa6040703b0 2024-01-27T23:24:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.545+0000 7fa5ebfff700 1 -- 172.21.15.121:0/48335562 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v5) v1 ==== 56+0+103 (secure 0 0 0) 0x7fa5f4010360 con 0x7fa6040703b0 2024-01-27T23:24:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-01-27T23:24:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 5, 2024-01-27T23:24:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": true, 2024-01-27T23:24:37.534 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "active_name": "smithi121.hcppvz", 2024-01-27T23:24:37.535 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-27T23:24:37.535 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 msgr2=0x7fa5f0032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 0x7fa5f0032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 msgr2=0x7fa60410cc10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa60410cc10 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa5f400b190 tx=0x7fa5f4006ee0).stop 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 shutdown_connections 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7fa5f002fd50 0x7fa5f0032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 --2- 172.21.15.121:0/48335562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa6040703b0 0x7fa60410cc10 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 >> 172.21.15.121:0/48335562 conn(0x7fa6040f63c0 msgr2=0x7fa6040f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 shutdown_connections 2024-01-27T23:24:37.538 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:37.548+0000 7fa60a253700 1 -- 172.21.15.121:0/48335562 wait complete. 2024-01-27T23:24:37.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:37 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:36.542431+0000 mon.smithi121 (mon.0) 26 : audit [INF] from='client.? 172.21.15.121:0/1633779619' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "cephadm"}]': finished 2024-01-27T23:24:37.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:37 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:36.542498+0000 mon.smithi121 (mon.0) 27 : cluster [DBG] mgrmap e5: smithi121.hcppvz(active, since 4s) 2024-01-27T23:24:37.919 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for the mgr to restart... 2024-01-27T23:24:37.919 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr epoch 5... 2024-01-27T23:24:37.919 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-27T23:24:38.306 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.316+0000 7f9f8e27d700 1 Processor -- start 2024-01-27T23:24:38.306 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.316+0000 7f9f8e27d700 1 -- start start 2024-01-27T23:24:38.306 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.316+0000 7f9f8e27d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f880fe940 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:38.306 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.316+0000 7f9f8e27d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f880fee30 con 0x7f9f880703d0 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.316+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f880fe940 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.316+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f880fe940 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53582/0 (socket says 172.21.15.121:53582) 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.316+0000 7f9f87fff700 1 -- 172.21.15.121:0/1257952566 learned_addr learned my addr 172.21.15.121:0/1257952566 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.317+0000 7f9f87fff700 1 -- 172.21.15.121:0/1257952566 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f880fef70 con 0x7f9f880703d0 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.317+0000 7f9f87fff700 1 --2- 172.21.15.121:0/1257952566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f880fe940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9f7800b5f0 tx=0x7f9f78006ee0).ready entity=mon.0 client_cookie=35e50479457e3234 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.317+0000 7f9f86ffd700 1 -- 172.21.15.121:0/1257952566 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f7800d600 con 0x7f9f880703d0 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.317+0000 7f9f86ffd700 1 -- 172.21.15.121:0/1257952566 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9f7800dbd0 con 0x7f9f880703d0 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.317+0000 7f9f86ffd700 1 -- 172.21.15.121:0/1257952566 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f78011c00 con 0x7f9f880703d0 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.318+0000 7f9f8e27d700 1 -- 172.21.15.121:0/1257952566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 msgr2=0x7f9f880fe940 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.318+0000 7f9f8e27d700 1 --2- 172.21.15.121:0/1257952566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f880fe940 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f9f7800b5f0 tx=0x7f9f78006ee0).stop 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.318+0000 7f9f8e27d700 1 -- 172.21.15.121:0/1257952566 shutdown_connections 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.318+0000 7f9f8e27d700 1 --2- 172.21.15.121:0/1257952566 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f880fe940 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:38.307 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.318+0000 7f9f8e27d700 1 -- 172.21.15.121:0/1257952566 >> 172.21.15.121:0/1257952566 conn(0x7f9f880f63c0 msgr2=0x7f9f880f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:38.308 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.318+0000 7f9f8e27d700 1 -- 172.21.15.121:0/1257952566 shutdown_connections 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.318+0000 7f9f8e27d700 1 -- 172.21.15.121:0/1257952566 wait complete. 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.319+0000 7f9f8e27d700 1 Processor -- start 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.319+0000 7f9f8e27d700 1 -- start start 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.319+0000 7f9f8e27d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f8810a9d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.319+0000 7f9f8e27d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9f8810aec0 con 0x7f9f880703d0 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.319+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f8810a9d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.319+0000 7f9f87fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f8810a9d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:53590/0 (socket says 172.21.15.121:53590) 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.319+0000 7f9f87fff700 1 -- 172.21.15.121:0/392760755 learned_addr learned my addr 172.21.15.121:0/392760755 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:38.309 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.320+0000 7f9f87fff700 1 -- 172.21.15.121:0/392760755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9f7800b200 con 0x7f9f880703d0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.320+0000 7f9f87fff700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f8810a9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9f7800b190 tx=0x7f9f78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.320+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f7801b410 con 0x7f9f880703d0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.320+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f9f7801b9e0 con 0x7f9f880703d0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.320+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9f7801a880 con 0x7f9f880703d0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.320+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9f8810b0c0 con 0x7f9f880703d0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.320+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9f8810b4d0 con 0x7f9f880703d0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.321+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 5) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f9f7800f070 con 0x7f9f880703d0 2024-01-27T23:24:38.310 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.321+0000 7f9f84ff9700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.321+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 --> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f88059100 con 0x7f9f7402fd50 2024-01-27T23:24:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.321+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(1..1 src has 1..1) v4 ==== 754+0+0 (secure 0 0 0) 0x7f9f78041200 con 0x7f9f880703d0 2024-01-27T23:24:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.321+0000 7f9f877fe700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 msgr2=0x7f9f74032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3662234193 2024-01-27T23:24:38.311 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.321+0000 7f9f877fe700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-27T23:24:38.510 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.522+0000 7f9f877fe700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 msgr2=0x7f9f74032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3662234193 2024-01-27T23:24:38.510 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.522+0000 7f9f877fe700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-27T23:24:38.836 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:37.545837+0000 mon.smithi121 (mon.0) 28 : audit [DBG] from='client.? 172.21.15.121:0/48335562' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-27T23:24:38.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.923+0000 7f9f877fe700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 msgr2=0x7f9f74032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3662234193 2024-01-27T23:24:38.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:38.923+0000 7f9f877fe700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-27T23:24:39.712 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:39.724+0000 7f9f877fe700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 msgr2=0x7f9f74032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3662234193 2024-01-27T23:24:39.712 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:39.724+0000 7f9f877fe700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-27T23:24:41.314 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:41.324+0000 7f9f877fe700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 msgr2=0x7f9f74032210 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/3662234193 2024-01-27T23:24:41.314 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:41.324+0000 7f9f877fe700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-27T23:24:42.238 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:42.248+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 6) v1 ==== 35970+0+0 (secure 0 0 0) 0x7f9f78022d80 con 0x7f9f880703d0 2024-01-27T23:24:42.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:42.248+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 msgr2=0x7f9f74032210 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T23:24:42.239 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:42.248+0000 7f9f84ff9700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:24:42.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:41.477040+0000 mon.smithi121 (mon.0) 29 : cluster [INF] Active manager daemon smithi121.hcppvz restarted 2024-01-27T23:24:42.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:41.477452+0000 mon.smithi121 (mon.0) 30 : cluster [INF] Activating manager daemon smithi121.hcppvz 2024-01-27T23:24:42.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:41.484051+0000 mon.smithi121 (mon.0) 31 : cluster [DBG] osdmap e2: 0 total, 0 up, 0 in 2024-01-27T23:24:43.240 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.250+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f9f7801b740 con 0x7f9f880703d0 2024-01-27T23:24:43.240 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.250+0000 7f9f84ff9700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f9f74033670 0x7f9f74035a60 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:43.240 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.250+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7f9f88059100 con 0x7f9f74033670 2024-01-27T23:24:43.241 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.251+0000 7f9f877fe700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f9f74033670 0x7f9f74035a60 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:43.241 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.251+0000 7f9f877fe700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f9f74033670 0x7f9f74035a60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9f7c006fd0 tx=0x7f9f7c006cf0).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:43.241 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.252+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7f9f88059100 con 0x7f9f74033670 2024-01-27T23:24:43.249 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.259+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7f9f88045ad0 con 0x7f9f74033670 2024-01-27T23:24:43.249 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.259+0000 7f9f84ff9700 1 -- 172.21.15.121:0/392760755 <== mgr.14114 v2:172.21.15.121:6800/1782949944 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+51 (secure 0 0 0) 0x7f9f88045ad0 con 0x7f9f74033670 2024-01-27T23:24:43.249 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-01-27T23:24:43.249 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap_epoch": 7, 2024-01-27T23:24:43.249 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "initialized": true 2024-01-27T23:24:43.249 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f9f74033670 msgr2=0x7f9f74035a60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f9f74033670 0x7f9f74035a60 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7f9f7c006fd0 tx=0x7f9f7c006cf0).stop 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 msgr2=0x7f9f8810a9d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f8810a9d0 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7f9f7800b190 tx=0x7f9f78006ee0).stop 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 shutdown_connections 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9f880703d0 0x7f9f8810a9d0 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f9f74033670 0x7f9f74035a60 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 --2- 172.21.15.121:0/392760755 >> [v2:172.21.15.121:6800/3662234193,v1:172.21.15.121:6801/3662234193] conn(0x7f9f7402fd50 0x7f9f74032210 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 >> 172.21.15.121:0/392760755 conn(0x7f9f880f63c0 msgr2=0x7f9f880f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 shutdown_connections 2024-01-27T23:24:43.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.260+0000 7f9f8e27d700 1 -- 172.21.15.121:0/392760755 wait complete. 2024-01-27T23:24:43.528 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:42.251043+0000 mon.smithi121 (mon.0) 32 : cluster [DBG] mgrmap e6: smithi121.hcppvz(active, starting, since 0.773659s) 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.252114+0000 mon.smithi121 (mon.0) 33 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.252852+0000 mon.smithi121 (mon.0) 34 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mgr metadata", "who": "smithi121.hcppvz", "id": "smithi121.hcppvz"}]: dispatch 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.253699+0000 mon.smithi121 (mon.0) 35 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.253880+0000 mon.smithi121 (mon.0) 36 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.254027+0000 mon.smithi121 (mon.0) 37 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:42.267824+0000 mon.smithi121 (mon.0) 38 : cluster [INF] Manager daemon smithi121.hcppvz is now available 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.283406+0000 mon.smithi121 (mon.0) 39 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.286587+0000 mon.smithi121 (mon.0) 40 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.289369+0000 mon.smithi121 (mon.0) 41 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.381820+0000 mon.smithi121 (mon.0) 42 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.382490+0000 mon.smithi121 (mon.0) 43 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.384058+0000 mon.smithi121 (mon.0) 44 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:24:43.529 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.385685+0000 mon.smithi121 (mon.0) 45 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:24:43.530 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.395827+0000 mon.smithi121 (mon.0) 46 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:24:43.530 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:42.396998+0000 mon.smithi121 (mon.0) 47 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:24:43.601 INFO:teuthology.orchestra.run.smithi121.stderr:mgr epoch 5 is available 2024-01-27T23:24:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:Setting orchestrator backend to cephadm... 2024-01-27T23:24:43.602 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch set backend cephadm 2024-01-27T23:24:43.956 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.963+0000 7f7a50a6a700 1 Processor -- start 2024-01-27T23:24:43.956 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.963+0000 7f7a50a6a700 1 -- start start 2024-01-27T23:24:43.956 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.963+0000 7f7a50a6a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:43.956 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.963+0000 7f7a50a6a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c0fc710 con 0x7f7a4c0703f0 2024-01-27T23:24:43.956 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.963+0000 7f7a4a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:43.956 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.963+0000 7f7a4a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54424/0 (socket says 172.21.15.121:54424) 2024-01-27T23:24:43.956 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.963+0000 7f7a4a59c700 1 -- 172.21.15.121:0/631054661 learned_addr learned my addr 172.21.15.121:0/631054661 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.964+0000 7f7a4a59c700 1 -- 172.21.15.121:0/631054661 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4c101210 con 0x7f7a4c0703f0 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.964+0000 7f7a4a59c700 1 --2- 172.21.15.121:0/631054661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c100d20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f7a4000b5f0 tx=0x7f7a40006ee0).ready entity=mon.0 client_cookie=32a4156c9a656e1b server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.964+0000 7f7a4959a700 1 -- 172.21.15.121:0/631054661 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4000d600 con 0x7f7a4c0703f0 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.964+0000 7f7a4959a700 1 -- 172.21.15.121:0/631054661 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7a4000dbd0 con 0x7f7a4c0703f0 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.964+0000 7f7a4959a700 1 -- 172.21.15.121:0/631054661 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a40011c00 con 0x7f7a4c0703f0 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.964+0000 7f7a50a6a700 1 -- 172.21.15.121:0/631054661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 msgr2=0x7f7a4c100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.964+0000 7f7a50a6a700 1 --2- 172.21.15.121:0/631054661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c100d20 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7f7a4000b5f0 tx=0x7f7a40006ee0).stop 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 -- 172.21.15.121:0/631054661 shutdown_connections 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 --2- 172.21.15.121:0/631054661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c100d20 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 -- 172.21.15.121:0/631054661 >> 172.21.15.121:0/631054661 conn(0x7f7a4c0f63c0 msgr2=0x7f7a4c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 -- 172.21.15.121:0/631054661 shutdown_connections 2024-01-27T23:24:43.957 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 -- 172.21.15.121:0/631054661 wait complete. 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 Processor -- start 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 -- start start 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c10cbc0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a50a6a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7a4c10d0b0 con 0x7f7a4c0703f0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a4a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c10cbc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a4a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c10cbc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54434/0 (socket says 172.21.15.121:54434) 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.965+0000 7f7a4a59c700 1 -- 172.21.15.121:0/4062028138 learned_addr learned my addr 172.21.15.121:0/4062028138 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.966+0000 7f7a4a59c700 1 -- 172.21.15.121:0/4062028138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a4000b200 con 0x7f7a4c0703f0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.966+0000 7f7a4a59c700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c10cbc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7a4000b190 tx=0x7f7a40006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.966+0000 7f7a3affd700 1 -- 172.21.15.121:0/4062028138 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4001b410 con 0x7f7a4c0703f0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.966+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7a4c10d2b0 con 0x7f7a4c0703f0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.966+0000 7f7a3affd700 1 -- 172.21.15.121:0/4062028138 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f7a4001b9e0 con 0x7f7a4c0703f0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.966+0000 7f7a3affd700 1 -- 172.21.15.121:0/4062028138 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a4001a880 con 0x7f7a4c0703f0 2024-01-27T23:24:43.958 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.966+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7a4c10d6c0 con 0x7f7a4c0703f0 2024-01-27T23:24:43.959 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.967+0000 7f7a3affd700 1 -- 172.21.15.121:0/4062028138 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 7) v1 ==== 36040+0+0 (secure 0 0 0) 0x7f7a4000f070 con 0x7f7a4c0703f0 2024-01-27T23:24:43.959 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.967+0000 7f7a3affd700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f7a3002fcf0 0x7f7a300321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:43.959 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.967+0000 7f7a3affd700 1 -- 172.21.15.121:0/4062028138 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f7a400410c0 con 0x7f7a4c0703f0 2024-01-27T23:24:43.959 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.967+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7a4c045ad0 con 0x7f7a4c0703f0 2024-01-27T23:24:43.959 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.967+0000 7f7a49d9b700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f7a3002fcf0 0x7f7a300321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:43.960 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.968+0000 7f7a49d9b700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f7a3002fcf0 0x7f7a300321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7a3c0096a0 tx=0x7f7a3c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:43.961 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:43.971+0000 7f7a3affd700 1 -- 172.21.15.121:0/4062028138 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a40025460 con 0x7f7a4c0703f0 2024-01-27T23:24:44.137 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.147+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}) v1 -- 0x7f7a4c106b60 con 0x7f7a3002fcf0 2024-01-27T23:24:44.142 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.152+0000 7f7a3affd700 1 -- 172.21.15.121:0/4062028138 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f7a4c106b60 con 0x7f7a3002fcf0 2024-01-27T23:24:44.144 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f7a3002fcf0 msgr2=0x7f7a300321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f7a3002fcf0 0x7f7a300321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f7a3c0096a0 tx=0x7f7a3c006b40).stop 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 msgr2=0x7f7a4c10cbc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c10cbc0 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7f7a4000b190 tx=0x7f7a40006ee0).stop 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 shutdown_connections 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f7a3002fcf0 0x7f7a300321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 --2- 172.21.15.121:0/4062028138 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7a4c0703f0 0x7f7a4c10cbc0 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 >> 172.21.15.121:0/4062028138 conn(0x7f7a4c0f63c0 msgr2=0x7f7a4c0f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 shutdown_connections 2024-01-27T23:24:44.145 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.154+0000 7f7a50a6a700 1 -- 172.21.15.121:0/4062028138 wait complete. 2024-01-27T23:24:44.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:43.253201+0000 mgr.smithi121.hcppvz (mgr.14114) 1 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-27T23:24:44.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:43.253990+0000 mon.smithi121 (mon.0) 48 : cluster [DBG] mgrmap e7: smithi121.hcppvz(active, since 1.7766s) 2024-01-27T23:24:44.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:43.260795+0000 mgr.smithi121.hcppvz (mgr.14114) 2 : audit [DBG] from='client.14118 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-27T23:24:44.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:44.153696+0000 mon.smithi121 (mon.0) 49 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:44.436 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:44.156848+0000 mon.smithi121 (mon.0) 50 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:24:44.501 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm set-user root 2024-01-27T23:24:44.837 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.845+0000 7f33e3fd4700 1 Processor -- start 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.845+0000 7f33e3fd4700 1 -- start start 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.845+0000 7f33e3fd4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc070a50 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.845+0000 7f33e3fd4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc0728d0 con 0x7f33dc0724b0 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e1d70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc070a50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e1d70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc070a50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54438/0 (socket says 172.21.15.121:54438) 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e1d70700 1 -- 172.21.15.121:0/1345562449 learned_addr learned my addr 172.21.15.121:0/1345562449 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e1d70700 1 -- 172.21.15.121:0/1345562449 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33dc070f40 con 0x7f33dc0724b0 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e1d70700 1 --2- 172.21.15.121:0/1345562449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc070a50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f33cc00b5f0 tx=0x7f33cc006ee0).ready entity=mon.0 client_cookie=1125b9bdc2f43e2a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e0d6e700 1 -- 172.21.15.121:0/1345562449 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc00d600 con 0x7f33dc0724b0 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e0d6e700 1 -- 172.21.15.121:0/1345562449 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f33cc00dbd0 con 0x7f33dc0724b0 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.846+0000 7f33e0d6e700 1 -- 172.21.15.121:0/1345562449 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc011c00 con 0x7f33dc0724b0 2024-01-27T23:24:44.838 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.847+0000 7f33e3fd4700 1 -- 172.21.15.121:0/1345562449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 msgr2=0x7f33dc070a50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.847+0000 7f33e3fd4700 1 --2- 172.21.15.121:0/1345562449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc070a50 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f33cc00b5f0 tx=0x7f33cc006ee0).stop 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.847+0000 7f33e3fd4700 1 -- 172.21.15.121:0/1345562449 shutdown_connections 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.847+0000 7f33e3fd4700 1 --2- 172.21.15.121:0/1345562449 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc070a50 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.847+0000 7f33e3fd4700 1 -- 172.21.15.121:0/1345562449 >> 172.21.15.121:0/1345562449 conn(0x7f33dc0f74c0 msgr2=0x7f33dc0f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.847+0000 7f33e3fd4700 1 -- 172.21.15.121:0/1345562449 shutdown_connections 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.847+0000 7f33e3fd4700 1 -- 172.21.15.121:0/1345562449 wait complete. 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e3fd4700 1 Processor -- start 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e3fd4700 1 -- start start 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e3fd4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc10cd00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e3fd4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f33dc10d1f0 con 0x7f33dc0724b0 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e1d70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc10cd00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e1d70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc10cd00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54444/0 (socket says 172.21.15.121:54444) 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e1d70700 1 -- 172.21.15.121:0/3218056698 learned_addr learned my addr 172.21.15.121:0/3218056698 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:44.839 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e1d70700 1 -- 172.21.15.121:0/3218056698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f33cc00b200 con 0x7f33dc0724b0 2024-01-27T23:24:44.840 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e1d70700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc10cd00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f33cc00e040 tx=0x7f33cc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33ca7fc700 1 -- 172.21.15.121:0/3218056698 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc00d9f0 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33ca7fc700 1 -- 172.21.15.121:0/3218056698 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f33cc01a9f0 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33ca7fc700 1 -- 172.21.15.121:0/3218056698 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f33cc01b8c0 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.848+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f33dc10d3f0 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.849+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f33dc10d800 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.849+0000 7f33ca7fc700 1 -- 172.21.15.121:0/3218056698 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f33cc00f070 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.849+0000 7f33ca7fc700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f33c402fdb0 0x7f33c4032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.849+0000 7f33ca7fc700 1 -- 172.21.15.121:0/3218056698 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f33cc0412f0 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.849+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f33dc045ad0 con 0x7f33dc0724b0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.849+0000 7f33e156f700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f33c402fdb0 0x7f33c4032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:44.841 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.850+0000 7f33e156f700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f33c402fdb0 0x7f33c4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f33d80096a0 tx=0x7f33d8006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:44.843 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:44.854+0000 7f33ca7fc700 1 -- 172.21.15.121:0/3218056698 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f33cc016070 con 0x7f33dc0724b0 2024-01-27T23:24:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.004+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}) v1 -- 0x7f33dc10df30 con 0x7f33c402fdb0 2024-01-27T23:24:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.005+0000 7f33ca7fc700 1 -- 172.21.15.121:0/3218056698 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+16 (secure 0 0 0) 0x7f33dc10df30 con 0x7f33c402fdb0 2024-01-27T23:24:44.994 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: value unchanged 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f33c402fdb0 msgr2=0x7f33c4032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f33c402fdb0 0x7f33c4032270 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f33d80096a0 tx=0x7f33d8006b40).stop 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 msgr2=0x7f33dc10cd00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc10cd00 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f33cc00e040 tx=0x7f33cc006ee0).stop 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 shutdown_connections 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f33c402fdb0 0x7f33c4032270 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 --2- 172.21.15.121:0/3218056698 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f33dc0724b0 0x7f33dc10cd00 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 >> 172.21.15.121:0/3218056698 conn(0x7f33dc0f74c0 msgr2=0x7f33dc0f9000 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.006+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 shutdown_connections 2024-01-27T23:24:44.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:45.007+0000 7f33e3fd4700 1 -- 172.21.15.121:0/3218056698 wait complete. 2024-01-27T23:24:45.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:45 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:44.149871+0000 mgr.smithi121.hcppvz (mgr.14114) 3 : audit [DBG] from='client.14124 -' entity='client.admin' cmd=[{"prefix": "orch set backend", "module_name": "cephadm", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:24:45.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:45 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:44.258676+0000 mon.smithi121 (mon.0) 51 : cluster [DBG] mgrmap e8: smithi121.hcppvz(active, since 2s) 2024-01-27T23:24:46.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:45.006177+0000 mgr.smithi121.hcppvz (mgr.14114) 4 : audit [DBG] from='client.14126 -' entity='client.admin' cmd=[{"prefix": "cephadm set-user", "user": "root", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:24:46.884 INFO:teuthology.orchestra.run.smithi121.stderr:Generating ssh key... 2024-01-27T23:24:46.884 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm generate-key 2024-01-27T23:24:48.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.064+0000 7f40b8cdf700 1 Processor -- start 2024-01-27T23:24:48.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.064+0000 7f40b8cdf700 1 -- start start 2024-01-27T23:24:48.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b8cdf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:48.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b8cdf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40fedd0 con 0x7f40b4070430 2024-01-27T23:24:48.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:48.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54452/0 (socket says 172.21.15.121:54452) 2024-01-27T23:24:48.297 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b259c700 1 -- 172.21.15.121:0/2929536600 learned_addr learned my addr 172.21.15.121:0/2929536600 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b259c700 1 -- 172.21.15.121:0/2929536600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40b40fef10 con 0x7f40b4070430 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b259c700 1 --2- 172.21.15.121:0/2929536600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fe8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f40a400b5f0 tx=0x7f40a4006ee0).ready entity=mon.0 client_cookie=fce32c6e596b38fa server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b159a700 1 -- 172.21.15.121:0/2929536600 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a400d600 con 0x7f40b4070430 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b159a700 1 -- 172.21.15.121:0/2929536600 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f40a400dbd0 con 0x7f40b4070430 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.065+0000 7f40b159a700 1 -- 172.21.15.121:0/2929536600 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a4011c00 con 0x7f40b4070430 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 -- 172.21.15.121:0/2929536600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 msgr2=0x7f40b40fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 --2- 172.21.15.121:0/2929536600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fe8e0 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f40a400b5f0 tx=0x7f40a4006ee0).stop 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 -- 172.21.15.121:0/2929536600 shutdown_connections 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 --2- 172.21.15.121:0/2929536600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fe8e0 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 -- 172.21.15.121:0/2929536600 >> 172.21.15.121:0/2929536600 conn(0x7f40b40f63c0 msgr2=0x7f40b40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 -- 172.21.15.121:0/2929536600 shutdown_connections 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 -- 172.21.15.121:0/2929536600 wait complete. 2024-01-27T23:24:48.298 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.066+0000 7f40b8cdf700 1 Processor -- start 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b8cdf700 1 -- start start 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b8cdf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fc360 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b8cdf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f40b40fa7e0 con 0x7f40b4070430 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fc360 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fc360 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:54458/0 (socket says 172.21.15.121:54458) 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b259c700 1 -- 172.21.15.121:0/900653366 learned_addr learned my addr 172.21.15.121:0/900653366 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b259c700 1 -- 172.21.15.121:0/900653366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f40a400b200 con 0x7f40b4070430 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b259c700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fc360 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f40a400b190 tx=0x7f40a4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40aaffd700 1 -- 172.21.15.121:0/900653366 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a401b410 con 0x7f40b4070430 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40aaffd700 1 -- 172.21.15.121:0/900653366 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f40a401b9e0 con 0x7f40b4070430 2024-01-27T23:24:48.299 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f40b40fa980 con 0x7f40b4070430 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40aaffd700 1 -- 172.21.15.121:0/900653366 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f40a401a880 con 0x7f40b4070430 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.067+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f40b40fad90 con 0x7f40b4070430 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.068+0000 7f40aaffd700 1 -- 172.21.15.121:0/900653366 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f40a400f070 con 0x7f40b4070430 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.068+0000 7f40aaffd700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f409402fdb0 0x7f4094032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.068+0000 7f40aaffd700 1 -- 172.21.15.121:0/900653366 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f40a40412b0 con 0x7f40b4070430 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.068+0000 7f40b1d9b700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f409402fdb0 0x7f4094032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.068+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f40b4059100 con 0x7f40b4070430 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.069+0000 7f40b1d9b700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f409402fdb0 0x7f4094032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.073+0000 7f40aaffd700 1 -- 172.21.15.121:0/900653366 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f40a4016070 con 0x7f40b4070430 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.224+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "cephadm generate-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f40b41048c0 con 0x7f409402fdb0 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.302+0000 7f40aaffd700 1 -- 172.21.15.121:0/900653366 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+0 (secure 0 0 0) 0x7f40b41048c0 con 0x7f409402fdb0 2024-01-27T23:24:48.302 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.303+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f409402fdb0 msgr2=0x7f4094032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.303+0000 7f40b8cdf700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f409402fdb0 0x7f4094032270 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f409c0096a0 tx=0x7f409c006b40).stop 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.303+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 msgr2=0x7f40b40fc360 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.303+0000 7f40b8cdf700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fc360 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7f40a400b190 tx=0x7f40a4006ee0).stop 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.303+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 shutdown_connections 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.303+0000 7f40b8cdf700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f409402fdb0 0x7f4094032270 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.304+0000 7f40b8cdf700 1 --2- 172.21.15.121:0/900653366 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f40b4070430 0x7f40b40fc360 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.304+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 >> 172.21.15.121:0/900653366 conn(0x7f40b40f63c0 msgr2=0x7f40b40f7f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.304+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 shutdown_connections 2024-01-27T23:24:48.303 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:48.304+0000 7f40b8cdf700 1 -- 172.21.15.121:0/900653366 wait complete. 2024-01-27T23:24:49.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:49 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:24:48.225853+0000 mgr.smithi121.hcppvz (mgr.14114) 6 : cephadm [INF] Generating ssh key... 2024-01-27T23:24:49.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:49 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:48.300267+0000 mon.smithi121 (mon.0) 52 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:49.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:49 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:48.302676+0000 mon.smithi121 (mon.0) 53 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:50.400 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 cephadm get-pub-key 2024-01-27T23:24:51.751 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d8c05a700 1 Processor -- start 2024-01-27T23:24:51.752 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d8c05a700 1 -- start start 2024-01-27T23:24:51.752 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d8c05a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d840fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d8c05a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d840fb490 con 0x7f4d840fab80 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d89df6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d840fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d89df6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d840fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45294/0 (socket says 172.21.15.121:45294) 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d89df6700 1 -- 172.21.15.121:0/3203033580 learned_addr learned my addr 172.21.15.121:0/3203033580 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.762+0000 7f4d89df6700 1 -- 172.21.15.121:0/3203033580 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d840fb5d0 con 0x7f4d840fab80 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d89df6700 1 --2- 172.21.15.121:0/3203033580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d840fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4d7800b5f0 tx=0x7f4d78006ee0).ready entity=mon.0 client_cookie=581127c33c4fe9f2 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d88df4700 1 -- 172.21.15.121:0/3203033580 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7800d600 con 0x7f4d840fab80 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d88df4700 1 -- 172.21.15.121:0/3203033580 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d7800dbd0 con 0x7f4d840fab80 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d88df4700 1 -- 172.21.15.121:0/3203033580 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d78011c00 con 0x7f4d840fab80 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d8c05a700 1 -- 172.21.15.121:0/3203033580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 msgr2=0x7f4d840fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:51.756 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d8c05a700 1 --2- 172.21.15.121:0/3203033580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d840fafa0 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f4d7800b5f0 tx=0x7f4d78006ee0).stop 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d8c05a700 1 -- 172.21.15.121:0/3203033580 shutdown_connections 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d8c05a700 1 --2- 172.21.15.121:0/3203033580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d840fafa0 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.763+0000 7f4d8c05a700 1 -- 172.21.15.121:0/3203033580 >> 172.21.15.121:0/3203033580 conn(0x7f4d840f63c0 msgr2=0x7f4d840f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.764+0000 7f4d8c05a700 1 -- 172.21.15.121:0/3203033580 shutdown_connections 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.764+0000 7f4d8c05a700 1 -- 172.21.15.121:0/3203033580 wait complete. 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.764+0000 7f4d8c05a700 1 Processor -- start 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.764+0000 7f4d8c05a700 1 -- start start 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d8c05a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d841087d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d8c05a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d84108cc0 con 0x7f4d840fab80 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d89df6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d841087d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d89df6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d841087d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45300/0 (socket says 172.21.15.121:45300) 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d89df6700 1 -- 172.21.15.121:0/1516553150 learned_addr learned my addr 172.21.15.121:0/1516553150 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d89df6700 1 -- 172.21.15.121:0/1516553150 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d7800b200 con 0x7f4d840fab80 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d89df6700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d841087d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4d7800b1d0 tx=0x7f4d78006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d767fc700 1 -- 172.21.15.121:0/1516553150 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7801b410 con 0x7f4d840fab80 2024-01-27T23:24:51.757 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d767fc700 1 -- 172.21.15.121:0/1516553150 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f4d7801b9e0 con 0x7f4d840fab80 2024-01-27T23:24:51.758 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d767fc700 1 -- 172.21.15.121:0/1516553150 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d7801a840 con 0x7f4d840fab80 2024-01-27T23:24:51.758 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d84108ec0 con 0x7f4d840fab80 2024-01-27T23:24:51.758 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.765+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d841092d0 con 0x7f4d840fab80 2024-01-27T23:24:51.758 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.766+0000 7f4d767fc700 1 -- 172.21.15.121:0/1516553150 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f4d7800f070 con 0x7f4d840fab80 2024-01-27T23:24:51.758 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.766+0000 7f4d767fc700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f4d6c031ff0 0x7f4d6c0344b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:51.758 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.766+0000 7f4d767fc700 1 -- 172.21.15.121:0/1516553150 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f4d78041260 con 0x7f4d840fab80 2024-01-27T23:24:51.759 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.766+0000 7f4d895f5700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f4d6c031ff0 0x7f4d6c0344b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:51.759 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.767+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d84059100 con 0x7f4d840fab80 2024-01-27T23:24:51.759 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.767+0000 7f4d895f5700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f4d6c031ff0 0x7f4d6c0344b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4d800096a0 tx=0x7f4d80006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:51.761 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.771+0000 7f4d767fc700 1 -- 172.21.15.121:0/1516553150 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d78016070 con 0x7f4d840fab80 2024-01-27T23:24:51.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.922+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}) v1 -- 0x7f4d841026d0 con 0x7f4d6c031ff0 2024-01-27T23:24:51.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.922+0000 7f4d767fc700 1 -- 172.21.15.121:0/1516553150 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+595 (secure 0 0 0) 0x7f4d841026d0 con 0x7f4d6c031ff0 2024-01-27T23:24:51.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ2TqBNFQhs8DwLKyxUMf7e4dZd3i+f0IdYVNEgtQjjuetSP+I9E3wInRGjmJqo2JfkJp9zFy/BpByCfdvH5yfJvZ1DIxpkq2mh8MzEpbhxwKczNFTKgzrGbhOi7GthKziXHX2GJ+TW46lDKqz9yrT1Eewv7p7FwvwWk0BDwYObwVZFZpTPugpbUVZq8tC76a3XSjmwTTF+hA+5LlCmd0VMADZbsUg+2zkU6UYapUPPpLxDArJRARIvRSapKz8yiajYUlbbMbf04Yl1hq4j9oPRArWHXACKZaLHDtfy1vCcNmr9vCC6QReEl6vccHq+mH8MAr1wyg3zaF5ZeA2HBUJp0FtgA8S5n1fZdHl0CkCGw4iH43XpwFAqLIbZHOVvglUjqe13Pt6F5VR6RQ/kY31UBDNMRT9qxkJtHBpw3Q0bD2hVfpxdgjekqH4FwhatRsQS8D0R/C+spgd1aYiK3nunG1/tIVLY7RTNRg20LwjiIEPTagwOe1IMdZiAvi5lus= ceph-2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:24:51.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f4d6c031ff0 msgr2=0x7f4d6c0344b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:51.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f4d6c031ff0 0x7f4d6c0344b0 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f4d800096a0 tx=0x7f4d80006b40).stop 2024-01-27T23:24:51.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 msgr2=0x7f4d841087d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:51.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d841087d0 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7f4d7800b1d0 tx=0x7f4d78006ee0).stop 2024-01-27T23:24:51.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 shutdown_connections 2024-01-27T23:24:51.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f4d6c031ff0 0x7f4d6c0344b0 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:51.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 --2- 172.21.15.121:0/1516553150 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d840fab80 0x7f4d841087d0 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:51.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 >> 172.21.15.121:0/1516553150 conn(0x7f4d840f63c0 msgr2=0x7f4d840ff910 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:51.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 shutdown_connections 2024-01-27T23:24:51.915 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:51.924+0000 7f4d8c05a700 1 -- 172.21.15.121:0/1516553150 wait complete. 2024-01-27T23:24:53.039 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:51.923670+0000 mgr.smithi121.hcppvz (mgr.14114) 7 : audit [DBG] from='client.14130 -' entity='client.admin' cmd=[{"prefix": "cephadm get-pub-key", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:24:53.800 INFO:teuthology.orchestra.run.smithi121.stderr:Wrote public SSH key to /home/ubuntu/cephtest/ceph.pub 2024-01-27T23:24:53.800 INFO:teuthology.orchestra.run.smithi121.stderr:Adding key to root@localhost authorized_keys... 2024-01-27T23:24:53.800 INFO:teuthology.orchestra.run.smithi121.stderr:Adding host smithi121... 2024-01-27T23:24:53.800 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch host add smithi121 172.21.15.121 2024-01-27T23:24:54.521 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.528+0000 7fe60f81a700 1 Processor -- start 2024-01-27T23:24:54.521 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.528+0000 7fe60f81a700 1 -- start start 2024-01-27T23:24:54.521 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe60f81a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe6080fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe60f81a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe6080fb490 con 0x7fe6080fab80 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe60d5b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe6080fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe60d5b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe6080fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45312/0 (socket says 172.21.15.121:45312) 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe60d5b6700 1 -- 172.21.15.121:0/2630046914 learned_addr learned my addr 172.21.15.121:0/2630046914 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe60d5b6700 1 -- 172.21.15.121:0/2630046914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe6080fb5d0 con 0x7fe6080fab80 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe60d5b6700 1 --2- 172.21.15.121:0/2630046914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe6080fafa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe5fc00b5f0 tx=0x7fe5fc006ee0).ready entity=mon.0 client_cookie=5a8738a6dd1538bb server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe5fbfff700 1 -- 172.21.15.121:0/2630046914 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5fc00d600 con 0x7fe6080fab80 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe5fbfff700 1 -- 172.21.15.121:0/2630046914 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe5fc00dbd0 con 0x7fe6080fab80 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.529+0000 7fe5fbfff700 1 -- 172.21.15.121:0/2630046914 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5fc011c00 con 0x7fe6080fab80 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 -- 172.21.15.121:0/2630046914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 msgr2=0x7fe6080fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 --2- 172.21.15.121:0/2630046914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe6080fafa0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7fe5fc00b5f0 tx=0x7fe5fc006ee0).stop 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 -- 172.21.15.121:0/2630046914 shutdown_connections 2024-01-27T23:24:54.522 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 --2- 172.21.15.121:0/2630046914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe6080fafa0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 -- 172.21.15.121:0/2630046914 >> 172.21.15.121:0/2630046914 conn(0x7fe6080f63c0 msgr2=0x7fe6080f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 -- 172.21.15.121:0/2630046914 shutdown_connections 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 -- 172.21.15.121:0/2630046914 wait complete. 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.530+0000 7fe60f81a700 1 Processor -- start 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60f81a700 1 -- start start 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60f81a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe60810d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60f81a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe608070450 con 0x7fe6080fab80 2024-01-27T23:24:54.523 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60d5b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe60810d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:54.524 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60d5b6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe60810d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45314/0 (socket says 172.21.15.121:45314) 2024-01-27T23:24:54.524 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60d5b6700 1 -- 172.21.15.121:0/2711301914 learned_addr learned my addr 172.21.15.121:0/2711301914 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:54.524 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60d5b6700 1 -- 172.21.15.121:0/2711301914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe5fc00b200 con 0x7fe6080fab80 2024-01-27T23:24:54.524 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60d5b6700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe60810d700 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe5fc00b1d0 tx=0x7fe5fc006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe5f9ffb700 1 -- 172.21.15.121:0/2711301914 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5fc01b410 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe5f9ffb700 1 -- 172.21.15.121:0/2711301914 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe5fc01b9e0 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe5f9ffb700 1 -- 172.21.15.121:0/2711301914 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe5fc01a840 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.531+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe608070650 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.532+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe60810ddb0 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.532+0000 7fe5f9ffb700 1 -- 172.21.15.121:0/2711301914 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fe5fc00f070 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.532+0000 7fe5f9ffb700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fe5f002fd50 0x7fe5f0032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.532+0000 7fe5f9ffb700 1 -- 172.21.15.121:0/2711301914 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fe5fc041280 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.533+0000 7fe60cdb5700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fe5f002fd50 0x7fe5f0032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.533+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe608059100 con 0x7fe6080fab80 2024-01-27T23:24:54.525 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.533+0000 7fe60cdb5700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fe5f002fd50 0x7fe5f0032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe6040096a0 tx=0x7fe604006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:54.526 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.537+0000 7fe5f9ffb700 1 -- 172.21.15.121:0/2711301914 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe5fc016080 con 0x7fe6080fab80 2024-01-27T23:24:54.697 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:54.707+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}) v1 -- 0x7fe6080fc9d0 con 0x7fe5f002fd50 2024-01-27T23:24:55.315 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Added host 'smithi121' 2024-01-27T23:24:55.316 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.324+0000 7fe5f9ffb700 1 -- 172.21.15.121:0/2711301914 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7fe6080fc9d0 con 0x7fe5f002fd50 2024-01-27T23:24:55.317 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fe5f002fd50 msgr2=0x7fe5f0032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:55.317 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fe5f002fd50 0x7fe5f0032210 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7fe6040096a0 tx=0x7fe604006b40).stop 2024-01-27T23:24:55.317 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 msgr2=0x7fe60810d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:55.317 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe60810d700 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7fe5fc00b1d0 tx=0x7fe5fc006ee0).stop 2024-01-27T23:24:55.317 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 shutdown_connections 2024-01-27T23:24:55.318 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fe5f002fd50 0x7fe5f0032210 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:55.318 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 --2- 172.21.15.121:0/2711301914 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6080fab80 0x7fe60810d700 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:55.318 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 >> 172.21.15.121:0/2711301914 conn(0x7fe6080f63c0 msgr2=0x7fe6080f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:55.318 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 shutdown_connections 2024-01-27T23:24:55.318 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:55.327+0000 7fe60f81a700 1 -- 172.21.15.121:0/2711301914 wait complete. 2024-01-27T23:24:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying mon service with default placement... 2024-01-27T23:24:55.690 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mon 2024-01-27T23:24:56.050 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.057+0000 7f64daf5b700 1 Processor -- start 2024-01-27T23:24:56.050 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.057+0000 7f64daf5b700 1 -- start start 2024-01-27T23:24:56.050 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.058+0000 7f64daf5b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d40fe900 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:56.050 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.058+0000 7f64daf5b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d40fedf0 con 0x7f64d4070430 2024-01-27T23:24:56.050 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.058+0000 7f64d8cf7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d40fe900 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:56.050 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.058+0000 7f64d8cf7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d40fe900 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45324/0 (socket says 172.21.15.121:45324) 2024-01-27T23:24:56.050 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.058+0000 7f64d8cf7700 1 -- 172.21.15.121:0/2880748111 learned_addr learned my addr 172.21.15.121:0/2880748111 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.058+0000 7f64d8cf7700 1 -- 172.21.15.121:0/2880748111 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64d40fef30 con 0x7f64d4070430 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.058+0000 7f64d8cf7700 1 --2- 172.21.15.121:0/2880748111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d40fe900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f64c800b5f0 tx=0x7f64c8006ee0).ready entity=mon.0 client_cookie=fc6c20514bcee11e server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64d37fe700 1 -- 172.21.15.121:0/2880748111 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c800d600 con 0x7f64d4070430 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64d37fe700 1 -- 172.21.15.121:0/2880748111 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f64c800dbd0 con 0x7f64d4070430 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64d37fe700 1 -- 172.21.15.121:0/2880748111 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c8011c00 con 0x7f64d4070430 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64daf5b700 1 -- 172.21.15.121:0/2880748111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 msgr2=0x7f64d40fe900 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64daf5b700 1 --2- 172.21.15.121:0/2880748111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d40fe900 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7f64c800b5f0 tx=0x7f64c8006ee0).stop 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64daf5b700 1 -- 172.21.15.121:0/2880748111 shutdown_connections 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64daf5b700 1 --2- 172.21.15.121:0/2880748111 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d40fe900 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64daf5b700 1 -- 172.21.15.121:0/2880748111 >> 172.21.15.121:0/2880748111 conn(0x7f64d40f63c0 msgr2=0x7f64d40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64daf5b700 1 -- 172.21.15.121:0/2880748111 shutdown_connections 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.059+0000 7f64daf5b700 1 -- 172.21.15.121:0/2880748111 wait complete. 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.060+0000 7f64daf5b700 1 Processor -- start 2024-01-27T23:24:56.051 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.060+0000 7f64daf5b700 1 -- start start 2024-01-27T23:24:56.052 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.060+0000 7f64daf5b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d410aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:56.052 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.060+0000 7f64daf5b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f64d410af40 con 0x7f64d4070430 2024-01-27T23:24:56.052 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.060+0000 7f64d8cf7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d410aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:56.052 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.060+0000 7f64d8cf7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d410aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45336/0 (socket says 172.21.15.121:45336) 2024-01-27T23:24:56.052 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.060+0000 7f64d8cf7700 1 -- 172.21.15.121:0/2664615995 learned_addr learned my addr 172.21.15.121:0/2664615995 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.061+0000 7f64d8cf7700 1 -- 172.21.15.121:0/2664615995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f64c800b200 con 0x7f64d4070430 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.061+0000 7f64d8cf7700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d410aa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f64c800e040 tx=0x7f64c8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.061+0000 7f64d17fa700 1 -- 172.21.15.121:0/2664615995 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c800d9f0 con 0x7f64d4070430 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.061+0000 7f64d17fa700 1 -- 172.21.15.121:0/2664615995 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f64c801a9f0 con 0x7f64d4070430 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.061+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f64d410b140 con 0x7f64d4070430 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.061+0000 7f64d17fa700 1 -- 172.21.15.121:0/2664615995 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f64c801b8c0 con 0x7f64d4070430 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.061+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f64d410b550 con 0x7f64d4070430 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.062+0000 7f64d17fa700 1 -- 172.21.15.121:0/2664615995 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f64c800f070 con 0x7f64d4070430 2024-01-27T23:24:56.053 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.062+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f64d4045ad0 con 0x7f64d4070430 2024-01-27T23:24:56.054 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.062+0000 7f64d17fa700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f64b802fd50 0x7f64b8032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:56.054 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.062+0000 7f64d17fa700 1 -- 172.21.15.121:0/2664615995 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f64c80412d0 con 0x7f64d4070430 2024-01-27T23:24:56.054 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.062+0000 7f64d3fff700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f64b802fd50 0x7f64b8032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:56.054 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.062+0000 7f64d3fff700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f64b802fd50 0x7f64b8032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64c00096a0 tx=0x7f64c0006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:56.055 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.066+0000 7f64d17fa700 1 -- 172.21.15.121:0/2664615995 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f64c8025460 con 0x7f64d4070430 2024-01-27T23:24:56.230 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.240+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}) v1 -- 0x7f64d410be90 con 0x7f64b802fd50 2024-01-27T23:24:56.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.244+0000 7f64d17fa700 1 -- 172.21.15.121:0/2664615995 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f64d410be90 con 0x7f64b802fd50 2024-01-27T23:24:56.234 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled mon update... 2024-01-27T23:24:56.236 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f64b802fd50 msgr2=0x7f64b8032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:56.236 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f64b802fd50 0x7f64b8032210 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f64c00096a0 tx=0x7f64c0006b40).stop 2024-01-27T23:24:56.236 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 msgr2=0x7f64d410aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d410aa50 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7f64c800e040 tx=0x7f64c8006ee0).stop 2024-01-27T23:24:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 shutdown_connections 2024-01-27T23:24:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f64b802fd50 0x7f64b8032210 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 --2- 172.21.15.121:0/2664615995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64d4070430 0x7f64d410aa50 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 >> 172.21.15.121:0/2664615995 conn(0x7f64d40f63c0 msgr2=0x7f64d40f7ef0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 shutdown_connections 2024-01-27T23:24:56.237 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.246+0000 7f64daf5b700 1 -- 172.21.15.121:0/2664615995 wait complete. 2024-01-27T23:24:56.509 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:54.709816+0000 mgr.smithi121.hcppvz (mgr.14114) 8 : audit [DBG] from='client.14132 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi121", "addr": "172.21.15.121", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:24:56.509 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:55.325369+0000 mon.smithi121 (mon.0) 54 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:56.509 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:24:55.325657+0000 mgr.smithi121.hcppvz (mgr.14114) 9 : cephadm [INF] Added host smithi121 2024-01-27T23:24:56.509 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:55.325890+0000 mon.smithi121 (mon.0) 55 : audit [DBG] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:24:56.509 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:56.245703+0000 mon.smithi121 (mon.0) 56 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:56.582 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying mgr service with default placement... 2024-01-27T23:24:56.582 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply mgr 2024-01-27T23:24:56.940 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.950+0000 7fa35759e700 1 Processor -- start 2024-01-27T23:24:56.944 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35759e700 1 -- start start 2024-01-27T23:24:56.944 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35759e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580fec60 con 0x7fa3580703f0 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45350/0 (socket says 172.21.15.121:45350) 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35659c700 1 -- 172.21.15.121:0/1087549827 learned_addr learned my addr 172.21.15.121:0/1087549827 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35659c700 1 -- 172.21.15.121:0/1087549827 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3580feda0 con 0x7fa3580703f0 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.951+0000 7fa35659c700 1 --2- 172.21.15.121:0/1087549827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fe770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa34000b5f0 tx=0x7fa340006ee0).ready entity=mon.0 client_cookie=eb80ad41ebde9ec8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35559a700 1 -- 172.21.15.121:0/1087549827 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34000d600 con 0x7fa3580703f0 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35559a700 1 -- 172.21.15.121:0/1087549827 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa34000dbd0 con 0x7fa3580703f0 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35559a700 1 -- 172.21.15.121:0/1087549827 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa340011c00 con 0x7fa3580703f0 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35759e700 1 -- 172.21.15.121:0/1087549827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 msgr2=0x7fa3580fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35759e700 1 --2- 172.21.15.121:0/1087549827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fe770 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7fa34000b5f0 tx=0x7fa340006ee0).stop 2024-01-27T23:24:56.945 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35759e700 1 -- 172.21.15.121:0/1087549827 shutdown_connections 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35759e700 1 --2- 172.21.15.121:0/1087549827 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fe770 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35759e700 1 -- 172.21.15.121:0/1087549827 >> 172.21.15.121:0/1087549827 conn(0x7fa3580f6210 msgr2=0x7fa3580f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35759e700 1 -- 172.21.15.121:0/1087549827 shutdown_connections 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.952+0000 7fa35759e700 1 -- 172.21.15.121:0/1087549827 wait complete. 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35759e700 1 Processor -- start 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35759e700 1 -- start start 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fc1f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35759e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3580fa670 con 0x7fa3580703f0 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fc1f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fc1f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45354/0 (socket says 172.21.15.121:45354) 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35659c700 1 -- 172.21.15.121:0/1932425439 learned_addr learned my addr 172.21.15.121:0/1932425439 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35659c700 1 -- 172.21.15.121:0/1932425439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa34000b200 con 0x7fa3580703f0 2024-01-27T23:24:56.946 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.953+0000 7fa35659c700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fc1f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa34000e040 tx=0x7fa340006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.954+0000 7fa34effd700 1 -- 172.21.15.121:0/1932425439 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34000d9f0 con 0x7fa3580703f0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.954+0000 7fa34effd700 1 -- 172.21.15.121:0/1932425439 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fa34001a9f0 con 0x7fa3580703f0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.954+0000 7fa34effd700 1 -- 172.21.15.121:0/1932425439 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa34001b8c0 con 0x7fa3580703f0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.954+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3580fa810 con 0x7fa3580703f0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.954+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3580fac20 con 0x7fa3580703f0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.955+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa358045ad0 con 0x7fa3580703f0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.955+0000 7fa34effd700 1 -- 172.21.15.121:0/1932425439 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fa34000f070 con 0x7fa3580703f0 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.955+0000 7fa34effd700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fa33802fdb0 0x7fa338032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:56.947 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.955+0000 7fa34effd700 1 -- 172.21.15.121:0/1932425439 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fa340040830 con 0x7fa3580703f0 2024-01-27T23:24:56.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.955+0000 7fa355d9b700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fa33802fdb0 0x7fa338032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:56.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.956+0000 7fa355d9b700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fa33802fdb0 0x7fa338032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa3480096a0 tx=0x7fa348006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:56.948 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:56.959+0000 7fa34effd700 1 -- 172.21.15.121:0/1932425439 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa34001ab60 con 0x7fa3580703f0 2024-01-27T23:24:57.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.135+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}) v1 -- 0x7fa3581047f0 con 0x7fa33802fdb0 2024-01-27T23:24:57.128 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.138+0000 7fa34effd700 1 -- 172.21.15.121:0/1932425439 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7fa3581047f0 con 0x7fa33802fdb0 2024-01-27T23:24:57.128 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled mgr update... 2024-01-27T23:24:57.130 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fa33802fdb0 msgr2=0x7fa338032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:57.130 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fa33802fdb0 0x7fa338032270 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7fa3480096a0 tx=0x7fa348006b40).stop 2024-01-27T23:24:57.130 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 msgr2=0x7fa3580fc1f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:57.131 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fc1f0 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7fa34000e040 tx=0x7fa340006ee0).stop 2024-01-27T23:24:57.131 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 shutdown_connections 2024-01-27T23:24:57.131 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fa33802fdb0 0x7fa338032270 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:57.131 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 --2- 172.21.15.121:0/1932425439 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3580703f0 0x7fa3580fc1f0 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:57.131 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 >> 172.21.15.121:0/1932425439 conn(0x7fa3580f6210 msgr2=0x7fa3580f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:57.131 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 shutdown_connections 2024-01-27T23:24:57.131 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.140+0000 7fa35759e700 1 -- 172.21.15.121:0/1932425439 wait complete. 2024-01-27T23:24:57.473 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying crash service with default placement... 2024-01-27T23:24:57.474 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply crash 2024-01-27T23:24:57.578 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:57 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:56.242727+0000 mgr.smithi121.hcppvz (mgr.14114) 10 : audit [DBG] from='client.14134 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:24:57.578 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:57 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:24:56.243451+0000 mgr.smithi121.hcppvz (mgr.14114) 11 : cephadm [INF] Saving service mon spec with placement count:5 2024-01-27T23:24:57.578 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:57 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:57.139487+0000 mon.smithi121 (mon.0) 57 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.809+0000 7f6c9e7e2700 1 Processor -- start 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.809+0000 7f6c9e7e2700 1 -- start start 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.809+0000 7f6c9e7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c980fafc0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.809+0000 7f6c9e7e2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c980fb4b0 con 0x7f6c980faba0 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.809+0000 7f6c97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c980fafc0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.810+0000 7f6c97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c980fafc0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45368/0 (socket says 172.21.15.121:45368) 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.810+0000 7f6c97fff700 1 -- 172.21.15.121:0/3032297151 learned_addr learned my addr 172.21.15.121:0/3032297151 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:57.803 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.810+0000 7f6c97fff700 1 -- 172.21.15.121:0/3032297151 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c980fb5f0 con 0x7f6c980faba0 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.810+0000 7f6c97fff700 1 --2- 172.21.15.121:0/3032297151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c980fafc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6c8800b5f0 tx=0x7f6c88006ee0).ready entity=mon.0 client_cookie=1f6ebebe419f0076 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.810+0000 7f6c977fe700 1 -- 172.21.15.121:0/3032297151 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8800d600 con 0x7f6c980faba0 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.810+0000 7f6c977fe700 1 -- 172.21.15.121:0/3032297151 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6c8800dbd0 con 0x7f6c980faba0 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.810+0000 7f6c977fe700 1 -- 172.21.15.121:0/3032297151 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c88011c00 con 0x7f6c980faba0 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/3032297151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 msgr2=0x7f6c980fafc0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 --2- 172.21.15.121:0/3032297151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c980fafc0 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f6c8800b5f0 tx=0x7f6c88006ee0).stop 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/3032297151 shutdown_connections 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 --2- 172.21.15.121:0/3032297151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c980fafc0 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/3032297151 >> 172.21.15.121:0/3032297151 conn(0x7f6c980f63c0 msgr2=0x7f6c980f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/3032297151 shutdown_connections 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/3032297151 wait complete. 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 Processor -- start 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.811+0000 7f6c9e7e2700 1 -- start start 2024-01-27T23:24:57.804 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c9e7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c9810d750 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c9e7e2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6c98070400 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c9810d750 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c97fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c9810d750 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45380/0 (socket says 172.21.15.121:45380) 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c97fff700 1 -- 172.21.15.121:0/183667555 learned_addr learned my addr 172.21.15.121:0/183667555 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c97fff700 1 -- 172.21.15.121:0/183667555 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6c8800b200 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c97fff700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c9810d750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6c8800e040 tx=0x7f6c88006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c957fa700 1 -- 172.21.15.121:0/183667555 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8801b410 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c957fa700 1 -- 172.21.15.121:0/183667555 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f6c8801b9e0 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c957fa700 1 -- 172.21.15.121:0/183667555 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6c8801a820 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6c98070600 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.812+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6c9810de00 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.813+0000 7f6c957fa700 1 -- 172.21.15.121:0/183667555 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7f6c8800f070 con 0x7f6c980faba0 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.813+0000 7f6c957fa700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f6c7802fdb0 0x7f6c78032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:57.805 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.813+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6c98045ad0 con 0x7f6c980faba0 2024-01-27T23:24:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.813+0000 7f6c957fa700 1 -- 172.21.15.121:0/183667555 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7f6c88041350 con 0x7f6c980faba0 2024-01-27T23:24:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.813+0000 7f6c8ffff700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f6c7802fdb0 0x7f6c78032270 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:57.806 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.814+0000 7f6c8ffff700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f6c7802fdb0 0x7f6c78032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6c800096a0 tx=0x7f6c80006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:57.807 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.817+0000 7f6c957fa700 1 -- 172.21.15.121:0/183667555 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6c88016080 con 0x7f6c980faba0 2024-01-27T23:24:57.982 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.992+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}) v1 -- 0x7f6c98057f10 con 0x7f6c7802fdb0 2024-01-27T23:24:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.996+0000 7f6c957fa700 1 -- 172.21.15.121:0/183667555 <== mgr.14114 v2:172.21.15.121:6800/1782949944 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+26 (secure 0 0 0) 0x7f6c98057f10 con 0x7f6c7802fdb0 2024-01-27T23:24:57.985 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled crash update... 2024-01-27T23:24:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f6c7802fdb0 msgr2=0x7f6c78032270 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f6c7802fdb0 0x7f6c78032270 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f6c800096a0 tx=0x7f6c80006b40).stop 2024-01-27T23:24:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 msgr2=0x7f6c9810d750 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c9810d750 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f6c8800e040 tx=0x7f6c88006ee0).stop 2024-01-27T23:24:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 shutdown_connections 2024-01-27T23:24:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7f6c7802fdb0 0x7f6c78032270 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:57.988 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 --2- 172.21.15.121:0/183667555 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6c980faba0 0x7f6c9810d750 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 >> 172.21.15.121:0/183667555 conn(0x7f6c980f63c0 msgr2=0x7f6c980f7ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 shutdown_connections 2024-01-27T23:24:57.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:57.998+0000 7f6c9e7e2700 1 -- 172.21.15.121:0/183667555 wait complete. 2024-01-27T23:24:58.331 INFO:teuthology.orchestra.run.smithi121.stderr:Enabling mgr prometheus module... 2024-01-27T23:24:58.332 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable prometheus 2024-01-27T23:24:58.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:57.136902+0000 mgr.smithi121.hcppvz (mgr.14114) 12 : audit [DBG] from='client.14136 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mgr", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:24:58.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:24:57.137399+0000 mgr.smithi121.hcppvz (mgr.14114) 13 : cephadm [INF] Saving service mgr spec with placement count:2 2024-01-27T23:24:58.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:57.997218+0000 mon.smithi121 (mon.0) 58 : audit [INF] from='mgr.14114 172.21.15.121:0/1029963063' entity='mgr.smithi121.hcppvz' 2024-01-27T23:24:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.688+0000 7fbe3e895700 1 Processor -- start 2024-01-27T23:24:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.688+0000 7fbe3e895700 1 -- start start 2024-01-27T23:24:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.688+0000 7fbe3e895700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe380fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.688+0000 7fbe3e895700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe380fb490 con 0x7fbe380fab80 2024-01-27T23:24:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.688+0000 7fbe37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe380fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:58.680 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.688+0000 7fbe37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe380fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45390/0 (socket says 172.21.15.121:45390) 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.688+0000 7fbe37fff700 1 -- 172.21.15.121:0/580770583 learned_addr learned my addr 172.21.15.121:0/580770583 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe37fff700 1 -- 172.21.15.121:0/580770583 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe380fb5d0 con 0x7fbe380fab80 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe37fff700 1 --2- 172.21.15.121:0/580770583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe380fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fbe2c00b5f0 tx=0x7fbe2c006ee0).ready entity=mon.0 client_cookie=17033ef550282cb9 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe36ffd700 1 -- 172.21.15.121:0/580770583 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c00d600 con 0x7fbe380fab80 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe36ffd700 1 -- 172.21.15.121:0/580770583 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe2c00dbd0 con 0x7fbe380fab80 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe36ffd700 1 -- 172.21.15.121:0/580770583 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c011c00 con 0x7fbe380fab80 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe3e895700 1 -- 172.21.15.121:0/580770583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 msgr2=0x7fbe380fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe3e895700 1 --2- 172.21.15.121:0/580770583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe380fafa0 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7fbe2c00b5f0 tx=0x7fbe2c006ee0).stop 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe3e895700 1 -- 172.21.15.121:0/580770583 shutdown_connections 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe3e895700 1 --2- 172.21.15.121:0/580770583 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe380fafa0 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe3e895700 1 -- 172.21.15.121:0/580770583 >> 172.21.15.121:0/580770583 conn(0x7fbe380f63c0 msgr2=0x7fbe380f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.689+0000 7fbe3e895700 1 -- 172.21.15.121:0/580770583 shutdown_connections 2024-01-27T23:24:58.681 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.690+0000 7fbe3e895700 1 -- 172.21.15.121:0/580770583 wait complete. 2024-01-27T23:24:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.690+0000 7fbe3e895700 1 Processor -- start 2024-01-27T23:24:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.690+0000 7fbe3e895700 1 -- start start 2024-01-27T23:24:58.682 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.690+0000 7fbe3e895700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe3810d700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.690+0000 7fbe3e895700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe38070450 con 0x7fbe380fab80 2024-01-27T23:24:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.690+0000 7fbe37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe3810d700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe3810d700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45400/0 (socket says 172.21.15.121:45400) 2024-01-27T23:24:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe37fff700 1 -- 172.21.15.121:0/1132311730 learned_addr learned my addr 172.21.15.121:0/1132311730 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:24:58.683 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe37fff700 1 -- 172.21.15.121:0/1132311730 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe2c00b200 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe37fff700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe3810d700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fbe2c00b1d0 tx=0x7fbe2c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c01b410 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe38070650 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fbe2c01b9e0 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe2c01a840 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.691+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe3810ddb0 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.692+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 8) v1 ==== 36120+0+0 (secure 0 0 0) 0x7fbe2c00f070 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.692+0000 7fbe34ff9700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fbe2402fd50 0x7fbe24032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.692+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fbe2c0413a0 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.692+0000 7fbe377fe700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fbe2402fd50 0x7fbe24032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.692+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe38059100 con 0x7fbe380fab80 2024-01-27T23:24:58.684 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.692+0000 7fbe377fe700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fbe2402fd50 0x7fbe24032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbe280096a0 tx=0x7fbe28006b40).ready entity=mgr.14114 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:24:58.685 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.696+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe2c016070 con 0x7fbe380fab80 2024-01-27T23:24:58.870 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:58.880+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "prometheus"} v 0) v1 -- 0x7fbe380fc960 con 0x7fbe380fab80 2024-01-27T23:24:59.319 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.329+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fbe2c0257e0 con 0x7fbe380fab80 2024-01-27T23:24:59.320 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.330+0000 7fbe34ff9700 1 -- 172.21.15.121:0/1132311730 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "prometheus"}]=0 v9) v1 ==== 89+0+0 (secure 0 0 0) 0x7fbe2c02e5e0 con 0x7fbe380fab80 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.332+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fbe2402fd50 msgr2=0x7fbe24032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.332+0000 7fbe3e895700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fbe2402fd50 0x7fbe24032210 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fbe280096a0 tx=0x7fbe28006b40).stop 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.332+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 msgr2=0x7fbe3810d700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.332+0000 7fbe3e895700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe3810d700 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7fbe2c00b1d0 tx=0x7fbe2c006ee0).stop 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.332+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 shutdown_connections 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.332+0000 7fbe3e895700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fbe2402fd50 0x7fbe24032210 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.333+0000 7fbe3e895700 1 --2- 172.21.15.121:0/1132311730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe380fab80 0x7fbe3810d700 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.333+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 >> 172.21.15.121:0/1132311730 conn(0x7fbe380f63c0 msgr2=0x7fbe380f7ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:24:59.323 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.333+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 shutdown_connections 2024-01-27T23:24:59.324 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:24:59.333+0000 7fbe3e895700 1 -- 172.21.15.121:0/1132311730 wait complete. 2024-01-27T23:24:59.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:59 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:57.994512+0000 mgr.smithi121.hcppvz (mgr.14114) 14 : audit [DBG] from='client.14138 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "crash", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:24:59.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:59 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:24:57.995136+0000 mgr.smithi121.hcppvz (mgr.14114) 15 : cephadm [INF] Saving service crash spec with placement * 2024-01-27T23:24:59.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:24:59 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:58.882195+0000 mon.smithi121 (mon.0) 59 : audit [INF] from='client.? 172.21.15.121:0/1132311730' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "prometheus"}]: dispatch 2024-01-27T23:24:59.705 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying prometheus service with default placement... 2024-01-27T23:24:59.706 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply prometheus 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc131bb9700 1 Processor -- start 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc131bb9700 1 -- start start 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc131bb9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c0fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc131bb9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0fb490 con 0x7fc12c0fab80 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc12b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c0fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc12b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c0fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45428/0 (socket says 172.21.15.121:45428) 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc12b7fe700 1 -- 172.21.15.121:0/3981968526 learned_addr learned my addr 172.21.15.121:0/3981968526 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.078+0000 7fc12b7fe700 1 -- 172.21.15.121:0/3981968526 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc12c0fb5d0 con 0x7fc12c0fab80 2024-01-27T23:25:00.069 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.079+0000 7fc12b7fe700 1 --2- 172.21.15.121:0/3981968526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c0fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc11c00b5f0 tx=0x7fc11c006ee0).ready entity=mon.0 client_cookie=42ad99369b96783f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:00.070 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.079+0000 7fc12a7fc700 1 -- 172.21.15.121:0/3981968526 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc11c00d600 con 0x7fc12c0fab80 2024-01-27T23:25:00.070 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.079+0000 7fc12a7fc700 1 -- 172.21.15.121:0/3981968526 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc11c00dbd0 con 0x7fc12c0fab80 2024-01-27T23:25:00.070 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.079+0000 7fc12a7fc700 1 -- 172.21.15.121:0/3981968526 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc11c011c00 con 0x7fc12c0fab80 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 -- 172.21.15.121:0/3981968526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 msgr2=0x7fc12c0fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 --2- 172.21.15.121:0/3981968526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c0fafa0 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fc11c00b5f0 tx=0x7fc11c006ee0).stop 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 -- 172.21.15.121:0/3981968526 shutdown_connections 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 --2- 172.21.15.121:0/3981968526 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c0fafa0 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 -- 172.21.15.121:0/3981968526 >> 172.21.15.121:0/3981968526 conn(0x7fc12c0f63c0 msgr2=0x7fc12c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 -- 172.21.15.121:0/3981968526 shutdown_connections 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 -- 172.21.15.121:0/3981968526 wait complete. 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 Processor -- start 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.080+0000 7fc131bb9700 1 -- start start 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc131bb9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c100e10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:00.072 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc131bb9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc12c0ff290 con 0x7fc12c0fab80 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc12b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c100e10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc12b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c100e10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45442/0 (socket says 172.21.15.121:45442) 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc12b7fe700 1 -- 172.21.15.121:0/1604284107 learned_addr learned my addr 172.21.15.121:0/1604284107 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc12b7fe700 1 -- 172.21.15.121:0/1604284107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc11c00b200 con 0x7fc12c0fab80 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc12b7fe700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c100e10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc11c00b1d0 tx=0x7fc11c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc11c01b410 con 0x7fc12c0fab80 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fc11c01b9e0 con 0x7fc12c0fab80 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc12c0ff430 con 0x7fc12c0fab80 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc11c01a840 con 0x7fc12c0fab80 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.081+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc12c0ff840 con 0x7fc12c0fab80 2024-01-27T23:25:00.073 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.082+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc12c045ad0 con 0x7fc12c0fab80 2024-01-27T23:25:00.074 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.083+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 9) v1 ==== 36134+0+0 (secure 0 0 0) 0x7fc11c00f070 con 0x7fc12c0fab80 2024-01-27T23:25:00.074 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.083+0000 7fc113fff700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:00.074 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.083+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(2..2 src has 1..2) v4 ==== 926+0+0 (secure 0 0 0) 0x7fc11c041320 con 0x7fc12c0fab80 2024-01-27T23:25:00.074 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.083+0000 7fc12affd700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 msgr2=0x7fc118032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1782949944 2024-01-27T23:25:00.074 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.083+0000 7fc12affd700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-27T23:25:00.076 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.087+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc11c019cd0 con 0x7fc12c0fab80 2024-01-27T23:25:00.260 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.271+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 --> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc12c0fca10 con 0x7fc11802fdb0 2024-01-27T23:25:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.284+0000 7fc12affd700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 msgr2=0x7fc118032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1782949944 2024-01-27T23:25:00.273 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.284+0000 7fc12affd700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-27T23:25:00.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:24:59.331994+0000 mon.smithi121 (mon.0) 60 : audit [INF] from='client.? 172.21.15.121:0/1132311730' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "prometheus"}]': finished 2024-01-27T23:25:00.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:24:59.332042+0000 mon.smithi121 (mon.0) 61 : cluster [DBG] mgrmap e9: smithi121.hcppvz(active, since 17s) 2024-01-27T23:25:00.674 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.685+0000 7fc12affd700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 msgr2=0x7fc118032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1782949944 2024-01-27T23:25:00.674 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:00.685+0000 7fc12affd700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-27T23:25:01.475 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:01.486+0000 7fc12affd700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 msgr2=0x7fc118032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1782949944 2024-01-27T23:25:01.475 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:01.486+0000 7fc12affd700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-27T23:25:03.076 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:03.087+0000 7fc12affd700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 msgr2=0x7fc118032270 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1782949944 2024-01-27T23:25:03.077 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:03.087+0000 7fc12affd700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-27T23:25:04.327 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:04.337+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 10) v1 ==== 35984+0+0 (secure 0 0 0) 0x7fc11c01bc10 con 0x7fc12c0fab80 2024-01-27T23:25:04.328 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:04.337+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 msgr2=0x7fc118032270 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T23:25:04.328 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:04.337+0000 7fc113fff700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:25:04.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:04.287140+0000 mon.smithi121 (mon.0) 62 : cluster [INF] Active manager daemon smithi121.hcppvz restarted 2024-01-27T23:25:04.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:04.287541+0000 mon.smithi121 (mon.0) 63 : cluster [INF] Activating manager daemon smithi121.hcppvz 2024-01-27T23:25:04.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:04.289852+0000 mon.smithi121 (mon.0) 64 : cluster [DBG] osdmap e3: 0 total, 0 up, 0 in 2024-01-27T23:25:05.335 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.344+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fc11c03d550 con 0x7fc12c0fab80 2024-01-27T23:25:05.335 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.344+0000 7fc113fff700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc118033c20 0x7fc118036010 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:05.335 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.344+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 --> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "prometheus", "target": ["mon-mgr", ""]}) v1 -- 0x7fc12c0fca10 con 0x7fc118033c20 2024-01-27T23:25:05.335 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.344+0000 7fc12affd700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc118033c20 0x7fc118036010 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:05.335 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.345+0000 7fc12affd700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc118033c20 0x7fc118036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc1200036d0 tx=0x7fc120006cb0).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:05.344 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.355+0000 7fc113fff700 1 -- 172.21.15.121:0/1604284107 <== mgr.14142 v2:172.21.15.121:6800/1334444756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+31 (secure 0 0 0) 0x7fc12c0fca10 con 0x7fc118033c20 2024-01-27T23:25:05.344 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled prometheus update... 2024-01-27T23:25:05.347 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc118033c20 msgr2=0x7fc118036010 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:05.347 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc118033c20 0x7fc118036010 secure :-1 s=READY pgs=1 cs=0 l=1 rev1=1 rx=0x7fc1200036d0 tx=0x7fc120006cb0).stop 2024-01-27T23:25:05.347 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 msgr2=0x7fc12c100e10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:05.347 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c100e10 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fc11c00b1d0 tx=0x7fc11c006ee0).stop 2024-01-27T23:25:05.347 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 shutdown_connections 2024-01-27T23:25:05.347 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc118033c20 0x7fc118036010 unknown :-1 s=CLOSED pgs=1 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:05.348 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc12c0fab80 0x7fc12c100e10 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:05.348 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 --2- 172.21.15.121:0/1604284107 >> [v2:172.21.15.121:6800/1782949944,v1:172.21.15.121:6801/1782949944] conn(0x7fc11802fdb0 0x7fc118032270 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:25:05.348 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 >> 172.21.15.121:0/1604284107 conn(0x7fc12c0f63c0 msgr2=0x7fc12c070030 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:05.348 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 shutdown_connections 2024-01-27T23:25:05.348 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:05.357+0000 7fc131bb9700 1 -- 172.21.15.121:0/1604284107 wait complete. 2024-01-27T23:25:05.682 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:04.339936+0000 mon.smithi121 (mon.0) 65 : cluster [DBG] mgrmap e10: smithi121.hcppvz(active, starting, since 0.0524999s) 2024-01-27T23:25:05.682 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.341083+0000 mon.smithi121 (mon.0) 66 : audit [DBG] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-01-27T23:25:05.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.341621+0000 mon.smithi121 (mon.0) 67 : audit [DBG] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mgr metadata", "who": "smithi121.hcppvz", "id": "smithi121.hcppvz"}]: dispatch 2024-01-27T23:25:05.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.342274+0000 mon.smithi121 (mon.0) 68 : audit [DBG] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-27T23:25:05.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.342402+0000 mon.smithi121 (mon.0) 69 : audit [DBG] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-27T23:25:05.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.342523+0000 mon.smithi121 (mon.0) 70 : audit [DBG] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-27T23:25:05.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:04.356658+0000 mon.smithi121 (mon.0) 71 : cluster [INF] Manager daemon smithi121.hcppvz is now available 2024-01-27T23:25:05.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.453695+0000 mon.smithi121 (mon.0) 72 : audit [DBG] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:25:05.683 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.454964+0000 mon.smithi121 (mon.0) 73 : audit [DBG] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:25:05.684 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.469361+0000 mon.smithi121 (mon.0) 74 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:25:05.684 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:04.477405+0000 mon.smithi121 (mon.0) 75 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:25:05.691 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying grafana service with default placement... 2024-01-27T23:25:05.691 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply grafana 2024-01-27T23:25:06.075 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.085+0000 7f99be328700 1 Processor -- start 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.085+0000 7f99be328700 1 -- start start 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.085+0000 7f99be328700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b80fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.085+0000 7f99be328700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b80fc5b0 con 0x7f99b80fbca0 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b80fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b80fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45198/0 (socket says 172.21.15.121:45198) 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b7fff700 1 -- 172.21.15.121:0/2919038011 learned_addr learned my addr 172.21.15.121:0/2919038011 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b7fff700 1 -- 172.21.15.121:0/2919038011 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b80fc6f0 con 0x7f99b80fbca0 2024-01-27T23:25:06.080 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b7fff700 1 --2- 172.21.15.121:0/2919038011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b80fc0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f99a800b5f0 tx=0x7f99a8006ee0).ready entity=mon.0 client_cookie=9caba05c9cae5e22 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b77fe700 1 -- 172.21.15.121:0/2919038011 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a800d600 con 0x7f99b80fbca0 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b77fe700 1 -- 172.21.15.121:0/2919038011 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f99a800dbd0 con 0x7f99b80fbca0 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.086+0000 7f99b77fe700 1 -- 172.21.15.121:0/2919038011 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a8011c00 con 0x7f99b80fbca0 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 -- 172.21.15.121:0/2919038011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 msgr2=0x7f99b80fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 --2- 172.21.15.121:0/2919038011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b80fc0c0 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7f99a800b5f0 tx=0x7f99a8006ee0).stop 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 -- 172.21.15.121:0/2919038011 shutdown_connections 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 --2- 172.21.15.121:0/2919038011 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b80fc0c0 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 -- 172.21.15.121:0/2919038011 >> 172.21.15.121:0/2919038011 conn(0x7f99b80f74c0 msgr2=0x7f99b80f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 -- 172.21.15.121:0/2919038011 shutdown_connections 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 -- 172.21.15.121:0/2919038011 wait complete. 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 Processor -- start 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.087+0000 7f99be328700 1 -- start start 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99be328700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b810aa50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99be328700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b810af40 con 0x7f99b80fbca0 2024-01-27T23:25:06.081 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b810aa50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b810aa50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45208/0 (socket says 172.21.15.121:45208) 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b7fff700 1 -- 172.21.15.121:0/1624273679 learned_addr learned my addr 172.21.15.121:0/1624273679 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b7fff700 1 -- 172.21.15.121:0/1624273679 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99a800b200 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b7fff700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b810aa50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f99a800e040 tx=0x7f99a8006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b57fa700 1 -- 172.21.15.121:0/1624273679 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a801b410 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b57fa700 1 -- 172.21.15.121:0/1624273679 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f99a801b9e0 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99b57fa700 1 -- 172.21.15.121:0/1624273679 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99a801a820 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b810b140 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.088+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b810b550 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.089+0000 7f99b57fa700 1 -- 172.21.15.121:0/1624273679 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7f99a800f070 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.089+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b8045ad0 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.089+0000 7f99b57fa700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f999802fcf0 0x7f99980321b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.089+0000 7f99b57fa700 1 -- 172.21.15.121:0/1624273679 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f99a8041420 con 0x7f99b80fbca0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.090+0000 7f99affff700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f999802fcf0 0x7f99980321b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:06.082 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.090+0000 7f99affff700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f999802fcf0 0x7f99980321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f99a00096a0 tx=0x7f99a0006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:06.083 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.094+0000 7f99b57fa700 1 -- 172.21.15.121:0/1624273679 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99a8019cd0 con 0x7f99b80fbca0 2024-01-27T23:25:06.260 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.270+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 --> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}) v1 -- 0x7f99b810be90 con 0x7f999802fcf0 2024-01-27T23:25:06.263 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.274+0000 7f99b57fa700 1 -- 172.21.15.121:0/1624273679 <== mgr.14142 v2:172.21.15.121:6800/1334444756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+28 (secure 0 0 0) 0x7f99b810be90 con 0x7f999802fcf0 2024-01-27T23:25:06.263 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled grafana update... 2024-01-27T23:25:06.265 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f999802fcf0 msgr2=0x7f99980321b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f999802fcf0 0x7f99980321b0 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7f99a00096a0 tx=0x7f99a0006b40).stop 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 msgr2=0x7f99b810aa50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b810aa50 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7f99a800e040 tx=0x7f99a8006ee0).stop 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 shutdown_connections 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f999802fcf0 0x7f99980321b0 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 --2- 172.21.15.121:0/1624273679 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b80fbca0 0x7f99b810aa50 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 >> 172.21.15.121:0/1624273679 conn(0x7f99b80f74c0 msgr2=0x7f99b80f8fc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 shutdown_connections 2024-01-27T23:25:06.266 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:06.275+0000 7f99be328700 1 -- 172.21.15.121:0/1624273679 wait complete. 2024-01-27T23:25:06.565 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:05.347715+0000 mon.smithi121 (mon.0) 76 : cluster [DBG] mgrmap e11: smithi121.hcppvz(active, since 1.06028s) 2024-01-27T23:25:06.565 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:25:05.348554+0000 mgr.smithi121.hcppvz (mgr.14142) 2 : cephadm [INF] Saving service prometheus spec with placement count:1 2024-01-27T23:25:06.565 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:05.355834+0000 mon.smithi121 (mon.0) 77 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:06.565 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:06.274871+0000 mon.smithi121 (mon.0) 78 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:06.612 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying node-exporter service with default placement... 2024-01-27T23:25:06.612 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply node-exporter 2024-01-27T23:25:07.006 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.015+0000 7fe1a81d2700 1 Processor -- start 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.015+0000 7fe1a81d2700 1 -- start start 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.015+0000 7fe1a81d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a00fe8e0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.015+0000 7fe1a81d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a00fedd0 con 0x7fe1a0070430 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.015+0000 7fe1a5f6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a00fe8e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.015+0000 7fe1a5f6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a00fe8e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45220/0 (socket says 172.21.15.121:45220) 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.015+0000 7fe1a5f6e700 1 -- 172.21.15.121:0/1856036702 learned_addr learned my addr 172.21.15.121:0/1856036702 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.016+0000 7fe1a5f6e700 1 -- 172.21.15.121:0/1856036702 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1a00fef10 con 0x7fe1a0070430 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.016+0000 7fe1a5f6e700 1 --2- 172.21.15.121:0/1856036702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a00fe8e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe19c00b5f0 tx=0x7fe19c006ee0).ready entity=mon.0 client_cookie=b1cf293c67fea491 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.016+0000 7fe1a4f6c700 1 -- 172.21.15.121:0/1856036702 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe19c00d600 con 0x7fe1a0070430 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.016+0000 7fe1a4f6c700 1 -- 172.21.15.121:0/1856036702 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe19c00dbd0 con 0x7fe1a0070430 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.016+0000 7fe1a4f6c700 1 -- 172.21.15.121:0/1856036702 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe19c011c00 con 0x7fe1a0070430 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.016+0000 7fe1a81d2700 1 -- 172.21.15.121:0/1856036702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 msgr2=0x7fe1a00fe8e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:07.007 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.016+0000 7fe1a81d2700 1 --2- 172.21.15.121:0/1856036702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a00fe8e0 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7fe19c00b5f0 tx=0x7fe19c006ee0).stop 2024-01-27T23:25:07.008 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.017+0000 7fe1a81d2700 1 -- 172.21.15.121:0/1856036702 shutdown_connections 2024-01-27T23:25:07.008 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.017+0000 7fe1a81d2700 1 --2- 172.21.15.121:0/1856036702 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a00fe8e0 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:07.008 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.017+0000 7fe1a81d2700 1 -- 172.21.15.121:0/1856036702 >> 172.21.15.121:0/1856036702 conn(0x7fe1a00f63c0 msgr2=0x7fe1a00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:07.008 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.017+0000 7fe1a81d2700 1 -- 172.21.15.121:0/1856036702 shutdown_connections 2024-01-27T23:25:07.008 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.017+0000 7fe1a81d2700 1 -- 172.21.15.121:0/1856036702 wait complete. 2024-01-27T23:25:07.009 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a81d2700 1 Processor -- start 2024-01-27T23:25:07.009 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a81d2700 1 -- start start 2024-01-27T23:25:07.009 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a81d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a010aa40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:07.009 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a81d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe1a010af30 con 0x7fe1a0070430 2024-01-27T23:25:07.009 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a5f6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a010aa40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:07.009 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a5f6e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a010aa40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45236/0 (socket says 172.21.15.121:45236) 2024-01-27T23:25:07.009 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a5f6e700 1 -- 172.21.15.121:0/3526259129 learned_addr learned my addr 172.21.15.121:0/3526259129 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a5f6e700 1 -- 172.21.15.121:0/3526259129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe19c00b200 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a5f6e700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a010aa40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe19c00e040 tx=0x7fe19c006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe18e7fc700 1 -- 172.21.15.121:0/3526259129 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe19c00d9f0 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe18e7fc700 1 -- 172.21.15.121:0/3526259129 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7fe19c01a9f0 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe1a010b130 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe18e7fc700 1 -- 172.21.15.121:0/3526259129 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe19c01b8c0 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.018+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe1a010b540 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.019+0000 7fe18e7fc700 1 -- 172.21.15.121:0/3526259129 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 11) v1 ==== 36054+0+0 (secure 0 0 0) 0x7fe19c00f070 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.019+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe1a0045ad0 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.019+0000 7fe18e7fc700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fe18802fd10 0x7fe1880321d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.019+0000 7fe18e7fc700 1 -- 172.21.15.121:0/3526259129 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fe19c041380 con 0x7fe1a0070430 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.020+0000 7fe1a576d700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fe18802fd10 0x7fe1880321d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:07.010 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.020+0000 7fe1a576d700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fe18802fd10 0x7fe1880321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe1900096a0 tx=0x7fe190006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:07.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.023+0000 7fe18e7fc700 1 -- 172.21.15.121:0/3526259129 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe19c022d70 con 0x7fe1a0070430 2024-01-27T23:25:07.185 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.196+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 --> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}) v1 -- 0x7fe1a010be80 con 0x7fe18802fd10 2024-01-27T23:25:07.189 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.200+0000 7fe18e7fc700 1 -- 172.21.15.121:0/3526259129 <== mgr.14142 v2:172.21.15.121:6800/1334444756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+34 (secure 0 0 0) 0x7fe1a010be80 con 0x7fe18802fd10 2024-01-27T23:25:07.189 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled node-exporter update... 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fe18802fd10 msgr2=0x7fe1880321d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fe18802fd10 0x7fe1880321d0 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7fe1900096a0 tx=0x7fe190006b40).stop 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 msgr2=0x7fe1a010aa40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a010aa40 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7fe19c00e040 tx=0x7fe19c006ee0).stop 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 shutdown_connections 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fe18802fd10 0x7fe1880321d0 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 --2- 172.21.15.121:0/3526259129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe1a0070430 0x7fe1a010aa40 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 >> 172.21.15.121:0/3526259129 conn(0x7fe1a00f63c0 msgr2=0x7fe1a00f7f60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 shutdown_connections 2024-01-27T23:25:07.192 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.201+0000 7fe1a81d2700 1 -- 172.21.15.121:0/3526259129 wait complete. 2024-01-27T23:25:07.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:07 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:06.272381+0000 mgr.smithi121.hcppvz (mgr.14142) 3 : audit [DBG] from='client.14150 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "grafana", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:25:07.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:07 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:25:06.272865+0000 mgr.smithi121.hcppvz (mgr.14142) 4 : cephadm [INF] Saving service grafana spec with placement count:1 2024-01-27T23:25:07.481 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:07 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:07.200850+0000 mon.smithi121 (mon.0) 79 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:07.482 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:07 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:07.276797+0000 mon.smithi121 (mon.0) 80 : cluster [DBG] mgrmap e12: smithi121.hcppvz(active, since 2s) 2024-01-27T23:25:07.562 INFO:teuthology.orchestra.run.smithi121.stderr:Deploying alertmanager service with default placement... 2024-01-27T23:25:07.563 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 orch apply alertmanager 2024-01-27T23:25:07.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.918+0000 7f2129c58700 1 Processor -- start 2024-01-27T23:25:07.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.918+0000 7f2129c58700 1 -- start start 2024-01-27T23:25:07.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f2129c58700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f21240fc0c0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:07.910 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f2129c58700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f21240fc5b0 con 0x7f21240fbca0 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f21240fc0c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f21240fc0c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45240/0 (socket says 172.21.15.121:45240) 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21237fe700 1 -- 172.21.15.121:0/4240406423 learned_addr learned my addr 172.21.15.121:0/4240406423 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21237fe700 1 -- 172.21.15.121:0/4240406423 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f21240fc6f0 con 0x7f21240fbca0 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21237fe700 1 --2- 172.21.15.121:0/4240406423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f21240fc0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f211800b5f0 tx=0x7f2118006ee0).ready entity=mon.0 client_cookie=99bf82d97155129 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21227fc700 1 -- 172.21.15.121:0/4240406423 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f211800d600 con 0x7f21240fbca0 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21227fc700 1 -- 172.21.15.121:0/4240406423 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f211800dbd0 con 0x7f21240fbca0 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.919+0000 7f21227fc700 1 -- 172.21.15.121:0/4240406423 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2118011c00 con 0x7f21240fbca0 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.920+0000 7f2129c58700 1 -- 172.21.15.121:0/4240406423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 msgr2=0x7f21240fc0c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.920+0000 7f2129c58700 1 --2- 172.21.15.121:0/4240406423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f21240fc0c0 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f211800b5f0 tx=0x7f2118006ee0).stop 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.920+0000 7f2129c58700 1 -- 172.21.15.121:0/4240406423 shutdown_connections 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.920+0000 7f2129c58700 1 --2- 172.21.15.121:0/4240406423 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f21240fc0c0 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.920+0000 7f2129c58700 1 -- 172.21.15.121:0/4240406423 >> 172.21.15.121:0/4240406423 conn(0x7f21240f74c0 msgr2=0x7f21240f98e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:07.911 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.920+0000 7f2129c58700 1 -- 172.21.15.121:0/4240406423 shutdown_connections 2024-01-27T23:25:07.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.920+0000 7f2129c58700 1 -- 172.21.15.121:0/4240406423 wait complete. 2024-01-27T23:25:07.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.921+0000 7f2129c58700 1 Processor -- start 2024-01-27T23:25:07.912 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.921+0000 7f2129c58700 1 -- start start 2024-01-27T23:25:07.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.921+0000 7f2129c58700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f212410aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:07.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.921+0000 7f2129c58700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f212410aef0 con 0x7f21240fbca0 2024-01-27T23:25:07.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.921+0000 7f21237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f212410aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:07.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.921+0000 7f21237fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f212410aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45248/0 (socket says 172.21.15.121:45248) 2024-01-27T23:25:07.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.921+0000 7f21237fe700 1 -- 172.21.15.121:0/2527899247 learned_addr learned my addr 172.21.15.121:0/2527899247 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:07.913 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.922+0000 7f21237fe700 1 -- 172.21.15.121:0/2527899247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f211800b200 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.922+0000 7f21237fe700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f212410aa00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f211800b1d0 tx=0x7f2118006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.922+0000 7f210bfff700 1 -- 172.21.15.121:0/2527899247 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f211801b410 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.922+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f212410b0f0 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.922+0000 7f210bfff700 1 -- 172.21.15.121:0/2527899247 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f211801b9e0 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.922+0000 7f210bfff700 1 -- 172.21.15.121:0/2527899247 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f211801a840 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.922+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f212410b500 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.923+0000 7f210bfff700 1 -- 172.21.15.121:0/2527899247 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f211800f070 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.923+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2124045ad0 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.923+0000 7f210bfff700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f211002fe60 0x7f2110032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.923+0000 7f210bfff700 1 -- 172.21.15.121:0/2527899247 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f21180414c0 con 0x7f21240fbca0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.923+0000 7f2122ffd700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f211002fe60 0x7f2110032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:07.914 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.923+0000 7f2122ffd700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f211002fe60 0x7f2110032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:07.917 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:07.927+0000 7f210bfff700 1 -- 172.21.15.121:0/2527899247 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2118022690 con 0x7f21240fbca0 2024-01-27T23:25:08.092 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.102+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 --> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}) v1 -- 0x7f212410be40 con 0x7f211002fe60 2024-01-27T23:25:08.095 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.106+0000 7f210bfff700 1 -- 172.21.15.121:0/2527899247 <== mgr.14142 v2:172.21.15.121:6800/1334444756 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+33 (secure 0 0 0) 0x7f212410be40 con 0x7f211002fe60 2024-01-27T23:25:08.096 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Scheduled alertmanager update... 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f211002fe60 msgr2=0x7f2110032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f211002fe60 0x7f2110032320 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7f21140096a0 tx=0x7f2114006b40).stop 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 msgr2=0x7f212410aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f212410aa00 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f211800b1d0 tx=0x7f2118006ee0).stop 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 shutdown_connections 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f211002fe60 0x7f2110032320 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 --2- 172.21.15.121:0/2527899247 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f21240fbca0 0x7f212410aa00 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:08.098 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 >> 172.21.15.121:0/2527899247 conn(0x7f21240f74c0 msgr2=0x7f21240f8f80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:08.099 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 shutdown_connections 2024-01-27T23:25:08.099 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.108+0000 7f2129c58700 1 -- 172.21.15.121:0/2527899247 wait complete. 2024-01-27T23:25:08.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:07.197883+0000 mgr.smithi121.hcppvz (mgr.14142) 5 : audit [DBG] from='client.14152 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "node-exporter", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:25:08.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:25:07.198545+0000 mgr.smithi121.hcppvz (mgr.14142) 6 : cephadm [INF] Saving service node-exporter spec with placement * 2024-01-27T23:25:08.414 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:08.107388+0000 mon.smithi121 (mon.0) 81 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:08.445 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/cephadm/container_init True --force 2024-01-27T23:25:08.823 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.830+0000 7f2631264700 1 Processor -- start 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.830+0000 7f2631264700 1 -- start start 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f2631264700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c0fe8f0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f2631264700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c0fede0 con 0x7f262c0704a0 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f262affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c0fe8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f262affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c0fe8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45254/0 (socket says 172.21.15.121:45254) 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f262affd700 1 -- 172.21.15.121:0/4077368191 learned_addr learned my addr 172.21.15.121:0/4077368191 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f262affd700 1 -- 172.21.15.121:0/4077368191 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262c0fef20 con 0x7f262c0704a0 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f262affd700 1 --2- 172.21.15.121:0/4077368191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c0fe8f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f262000b5f0 tx=0x7f2620006ee0).ready entity=mon.0 client_cookie=d1b34410d9a9c781 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f2629ffb700 1 -- 172.21.15.121:0/4077368191 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f262000d600 con 0x7f262c0704a0 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f2629ffb700 1 -- 172.21.15.121:0/4077368191 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f262000dbd0 con 0x7f262c0704a0 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.831+0000 7f2629ffb700 1 -- 172.21.15.121:0/4077368191 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2620011c00 con 0x7f262c0704a0 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.832+0000 7f2631264700 1 -- 172.21.15.121:0/4077368191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 msgr2=0x7f262c0fe8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:08.824 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.832+0000 7f2631264700 1 --2- 172.21.15.121:0/4077368191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c0fe8f0 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7f262000b5f0 tx=0x7f2620006ee0).stop 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.832+0000 7f2631264700 1 -- 172.21.15.121:0/4077368191 shutdown_connections 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.832+0000 7f2631264700 1 --2- 172.21.15.121:0/4077368191 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c0fe8f0 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.832+0000 7f2631264700 1 -- 172.21.15.121:0/4077368191 >> 172.21.15.121:0/4077368191 conn(0x7f262c0f63b0 msgr2=0x7f262c0f87d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.832+0000 7f2631264700 1 -- 172.21.15.121:0/4077368191 shutdown_connections 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.832+0000 7f2631264700 1 -- 172.21.15.121:0/4077368191 wait complete. 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f2631264700 1 Processor -- start 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f2631264700 1 -- start start 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f2631264700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c10aa00 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f2631264700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f262c10aef0 con 0x7f262c0704a0 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f262affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c10aa00 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f262affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c10aa00 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45258/0 (socket says 172.21.15.121:45258) 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f262affd700 1 -- 172.21.15.121:0/2961589387 learned_addr learned my addr 172.21.15.121:0/2961589387 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f262affd700 1 -- 172.21.15.121:0/2961589387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f262000b200 con 0x7f262c0704a0 2024-01-27T23:25:08.825 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f262affd700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c10aa00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f262000b190 tx=0x7f2620006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f26137fe700 1 -- 172.21.15.121:0/2961589387 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f262001b410 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f26137fe700 1 -- 172.21.15.121:0/2961589387 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1010+0+0 (secure 0 0 0) 0x7f262001b9e0 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f262c10b0f0 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f26137fe700 1 -- 172.21.15.121:0/2961589387 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f262001a880 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.833+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f262c10b500 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.834+0000 7f26137fe700 1 -- 172.21.15.121:0/2961589387 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f262000f070 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.834+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f262c045ad0 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.834+0000 7f26137fe700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f261802fe60 0x7f2618032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.834+0000 7f26137fe700 1 -- 172.21.15.121:0/2961589387 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f2620041570 con 0x7f262c0704a0 2024-01-27T23:25:08.826 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.834+0000 7f262a7fc700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f261802fe60 0x7f2618032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:08.827 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.835+0000 7f262a7fc700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f261802fe60 0x7f2618032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:08.828 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.838+0000 7f26137fe700 1 -- 172.21.15.121:0/2961589387 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2620010360 con 0x7f262c0704a0 2024-01-27T23:25:08.980 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.991+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/container_init}] v 0) v1 -- 0x7f262c0ffba0 con 0x7f262c0704a0 2024-01-27T23:25:08.984 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.995+0000 7f26137fe700 1 -- 172.21.15.121:0/2961589387 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/container_init}]=0 v8)=0 v8) v1 ==== 142+0+0 (secure 0 0 0) 0x7f2620016030 con 0x7f262c0704a0 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f261802fe60 msgr2=0x7f2618032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f261802fe60 0x7f2618032320 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7f261c0096a0 tx=0x7f261c006b40).stop 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 msgr2=0x7f262c10aa00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c10aa00 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f262000b190 tx=0x7f2620006ee0).stop 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 shutdown_connections 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f261802fe60 0x7f2618032320 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 --2- 172.21.15.121:0/2961589387 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f262c0704a0 0x7f262c10aa00 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 >> 172.21.15.121:0/2961589387 conn(0x7f262c0f63b0 msgr2=0x7f262c0f7f00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 shutdown_connections 2024-01-27T23:25:08.989 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:08.997+0000 7f2631264700 1 -- 172.21.15.121:0/2961589387 wait complete. 2024-01-27T23:25:09.547 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:08.104784+0000 mgr.smithi121.hcppvz (mgr.14142) 7 : audit [DBG] from='client.14154 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "alertmanager", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:25:09.547 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:25:08.105285+0000 mgr.smithi121.hcppvz (mgr.14142) 8 : cephadm [INF] Saving service alertmanager spec with placement count:1 2024-01-27T23:25:09.547 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:08.996404+0000 mon.smithi121 (mon.0) 82 : audit [INF] from='client.? 172.21.15.121:0/2961589387' entity='client.admin' 2024-01-27T23:25:09.595 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config set mgr mgr/dashboard/ssl_server_port 8443 2024-01-27T23:25:10.123 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.131+0000 7f7db4da6700 1 Processor -- start 2024-01-27T23:25:10.123 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.131+0000 7f7db4da6700 1 -- start start 2024-01-27T23:25:10.123 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7db4da6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db00fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:10.123 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7db4da6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db00fb490 con 0x7f7db00fab80 2024-01-27T23:25:10.123 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dae59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db00fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dae59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db00fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45264/0 (socket says 172.21.15.121:45264) 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dae59c700 1 -- 172.21.15.121:0/1328685529 learned_addr learned my addr 172.21.15.121:0/1328685529 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dae59c700 1 -- 172.21.15.121:0/1328685529 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7db00fb5d0 con 0x7f7db00fab80 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dae59c700 1 --2- 172.21.15.121:0/1328685529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db00fafa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7da000b5f0 tx=0x7f7da0006ee0).ready entity=mon.0 client_cookie=1b4d4acff1a06dc8 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dad59a700 1 -- 172.21.15.121:0/1328685529 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7da000d600 con 0x7f7db00fab80 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dad59a700 1 -- 172.21.15.121:0/1328685529 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7da000dc10 con 0x7f7db00fab80 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.132+0000 7f7dad59a700 1 -- 172.21.15.121:0/1328685529 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7da0011cd0 con 0x7f7db00fab80 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.133+0000 7f7db4da6700 1 -- 172.21.15.121:0/1328685529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 msgr2=0x7f7db00fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.133+0000 7f7db4da6700 1 --2- 172.21.15.121:0/1328685529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db00fafa0 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7f7da000b5f0 tx=0x7f7da0006ee0).stop 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.133+0000 7f7db4da6700 1 -- 172.21.15.121:0/1328685529 shutdown_connections 2024-01-27T23:25:10.124 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.133+0000 7f7db4da6700 1 --2- 172.21.15.121:0/1328685529 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db00fafa0 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:10.125 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.133+0000 7f7db4da6700 1 -- 172.21.15.121:0/1328685529 >> 172.21.15.121:0/1328685529 conn(0x7f7db00f63c0 msgr2=0x7f7db00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:10.125 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.133+0000 7f7db4da6700 1 -- 172.21.15.121:0/1328685529 shutdown_connections 2024-01-27T23:25:10.125 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.133+0000 7f7db4da6700 1 -- 172.21.15.121:0/1328685529 wait complete. 2024-01-27T23:25:10.125 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7db4da6700 1 Processor -- start 2024-01-27T23:25:10.125 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7db4da6700 1 -- start start 2024-01-27T23:25:10.125 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7db4da6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db010cd40 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:10.125 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7db4da6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7db010d230 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7dae59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db010cd40 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7dae59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db010cd40 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45266/0 (socket says 172.21.15.121:45266) 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7dae59c700 1 -- 172.21.15.121:0/3703791944 learned_addr learned my addr 172.21.15.121:0/3703791944 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7dae59c700 1 -- 172.21.15.121:0/3703791944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7da000b200 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.134+0000 7f7dae59c700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db010cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7da000dfb0 tx=0x7f7da0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.135+0000 7f7d9effd700 1 -- 172.21.15.121:0/3703791944 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7da0011df0 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.135+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7db010d430 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.135+0000 7f7d9effd700 1 -- 172.21.15.121:0/3703791944 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7da001b8c0 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.135+0000 7f7d9effd700 1 -- 172.21.15.121:0/3703791944 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7da001a8c0 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.135+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7db010d840 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.135+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7db0045ad0 con 0x7f7db00fab80 2024-01-27T23:25:10.126 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.136+0000 7f7d9effd700 1 -- 172.21.15.121:0/3703791944 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f7da000f070 con 0x7f7db00fab80 2024-01-27T23:25:10.127 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.136+0000 7f7d9effd700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f7d9402fe60 0x7f7d94032320 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:10.127 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.136+0000 7f7d9effd700 1 -- 172.21.15.121:0/3703791944 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f7da00417b0 con 0x7f7db00fab80 2024-01-27T23:25:10.127 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.136+0000 7f7dadd9b700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f7d9402fe60 0x7f7d94032320 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:10.127 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.137+0000 7f7dadd9b700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f7d9402fe60 0x7f7d94032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7da40096a0 tx=0x7f7da4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:10.129 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.140+0000 7f7d9effd700 1 -- 172.21.15.121:0/3703791944 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7da0022d70 con 0x7f7db00fab80 2024-01-27T23:25:10.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.294+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/dashboard/ssl_server_port}] v 0) v1 -- 0x7f7db00fc900 con 0x7f7db00fab80 2024-01-27T23:25:10.289 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.299+0000 7f7d9effd700 1 -- 172.21.15.121:0/3703791944 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/dashboard/ssl_server_port}]=0 v10)=0 v10) v1 ==== 130+0+0 (secure 0 0 0) 0x7f7da0016020 con 0x7f7db00fab80 2024-01-27T23:25:10.291 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f7d9402fe60 msgr2=0x7f7d94032320 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:10.291 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f7d9402fe60 0x7f7d94032320 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f7da40096a0 tx=0x7f7da4006b40).stop 2024-01-27T23:25:10.291 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 msgr2=0x7f7db010cd40 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:10.292 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db010cd40 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7f7da000dfb0 tx=0x7f7da0006ee0).stop 2024-01-27T23:25:10.292 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 shutdown_connections 2024-01-27T23:25:10.292 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f7d9402fe60 0x7f7d94032320 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:10.292 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 --2- 172.21.15.121:0/3703791944 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7db00fab80 0x7f7db010cd40 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:10.292 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 >> 172.21.15.121:0/3703791944 conn(0x7f7db00f63c0 msgr2=0x7f7db0070010 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:10.292 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 shutdown_connections 2024-01-27T23:25:10.292 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:10.301+0000 7f7db4da6700 1 -- 172.21.15.121:0/3703791944 wait complete. 2024-01-27T23:25:10.645 INFO:teuthology.orchestra.run.smithi121.stderr:Enabling the dashboard module... 2024-01-27T23:25:10.645 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr module enable dashboard 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.021+0000 7f5eeb0dc700 1 Processor -- start 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.021+0000 7f5eeb0dc700 1 -- start start 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.021+0000 7f5eeb0dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee40fe770 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.021+0000 7f5eeb0dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee40fec60 con 0x7f5ee40703f0 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.021+0000 7f5eea0da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee40fe770 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.021+0000 7f5eea0da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee40fe770 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45274/0 (socket says 172.21.15.121:45274) 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.021+0000 7f5eea0da700 1 -- 172.21.15.121:0/4053881586 learned_addr learned my addr 172.21.15.121:0/4053881586 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.022+0000 7f5eea0da700 1 -- 172.21.15.121:0/4053881586 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee40feda0 con 0x7f5ee40703f0 2024-01-27T23:25:11.013 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.022+0000 7f5eea0da700 1 --2- 172.21.15.121:0/4053881586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee40fe770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5ee000b5f0 tx=0x7f5ee0006ee0).ready entity=mon.0 client_cookie=704943d372e4910f server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.022+0000 7f5ee90d8700 1 -- 172.21.15.121:0/4053881586 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee000d600 con 0x7f5ee40703f0 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.022+0000 7f5ee90d8700 1 -- 172.21.15.121:0/4053881586 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ee000dc10 con 0x7f5ee40703f0 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.022+0000 7f5ee90d8700 1 -- 172.21.15.121:0/4053881586 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee0011cd0 con 0x7f5ee40703f0 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.022+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/4053881586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 msgr2=0x7f5ee40fe770 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.022+0000 7f5eeb0dc700 1 --2- 172.21.15.121:0/4053881586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee40fe770 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f5ee000b5f0 tx=0x7f5ee0006ee0).stop 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/4053881586 shutdown_connections 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 --2- 172.21.15.121:0/4053881586 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee40fe770 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/4053881586 >> 172.21.15.121:0/4053881586 conn(0x7f5ee40f6210 msgr2=0x7f5ee40f8630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/4053881586 shutdown_connections 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/4053881586 wait complete. 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 Processor -- start 2024-01-27T23:25:11.014 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 -- start start 2024-01-27T23:25:11.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee410a8f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:11.015 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eeb0dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ee410ade0 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eea0da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee410a8f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eea0da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee410a8f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:45278/0 (socket says 172.21.15.121:45278) 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.023+0000 7f5eea0da700 1 -- 172.21.15.121:0/3521685137 learned_addr learned my addr 172.21.15.121:0/3521685137 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.024+0000 7f5eea0da700 1 -- 172.21.15.121:0/3521685137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5ee000b200 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.024+0000 7f5eea0da700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee410a8f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5ee000df90 tx=0x7f5ee0006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.024+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee0011da0 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.024+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5ee001b8c0 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.024+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ee410afe0 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.024+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5ee001a930 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.024+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ee410b3f0 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.025+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 12) v1 ==== 36195+0+0 (secure 0 0 0) 0x7f5ee000f070 con 0x7f5ee40703f0 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.025+0000 7f5ed2ffd700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f5ecc02feb0 0x7f5ecc032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:11.017 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.025+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7f5ee0040fe0 con 0x7f5ee40703f0 2024-01-27T23:25:11.018 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.025+0000 7f5ee98d9700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f5ecc02feb0 0x7f5ecc032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:11.018 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.025+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ee4059100 con 0x7f5ee40703f0 2024-01-27T23:25:11.018 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.025+0000 7f5ee98d9700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f5ecc02feb0 0x7f5ecc032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5ed40096a0 tx=0x7f5ed4006b40).ready entity=mgr.14142 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:11.018 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.029+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5ee0016070 con 0x7f5ee40703f0 2024-01-27T23:25:11.213 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.224+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr module enable", "module": "dashboard"} v 0) v1 -- 0x7f5ee40ffa90 con 0x7f5ee40703f0 2024-01-27T23:25:11.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:09.930173+0000 mon.smithi121 (mon.0) 83 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:11.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:10.156905+0000 mon.smithi121 (mon.0) 84 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:11.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:10.300617+0000 mon.smithi121 (mon.0) 85 : audit [INF] from='client.? 172.21.15.121:0/3703791944' entity='client.admin' 2024-01-27T23:25:11.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.932+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f5ee0024740 con 0x7f5ee40703f0 2024-01-27T23:25:11.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.933+0000 7f5ed2ffd700 1 -- 172.21.15.121:0/3521685137 <== mon.0 v2:172.21.15.121:3300/0 8 ==== mon_command_ack([{"prefix": "mgr module enable", "module": "dashboard"}]=0 v13) v1 ==== 88+0+0 (secure 0 0 0) 0x7f5ee0005700 con 0x7f5ee40703f0 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f5ecc02feb0 msgr2=0x7f5ecc032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f5ecc02feb0 0x7f5ecc032370 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f5ed40096a0 tx=0x7f5ed4006b40).stop 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 msgr2=0x7f5ee410a8f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee410a8f0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f5ee000df90 tx=0x7f5ee0006ee0).stop 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 shutdown_connections 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7f5ecc02feb0 0x7f5ecc032370 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 --2- 172.21.15.121:0/3521685137 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ee40703f0 0x7f5ee410a8f0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:11.925 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 >> 172.21.15.121:0/3521685137 conn(0x7f5ee40f6210 msgr2=0x7f5ee40f7cd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 shutdown_connections 2024-01-27T23:25:11.926 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:11.935+0000 7f5eeb0dc700 1 -- 172.21.15.121:0/3521685137 wait complete. 2024-01-27T23:25:12.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:11 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:11.225869+0000 mon.smithi121 (mon.0) 86 : audit [INF] from='client.? 172.21.15.121:0/3521685137' entity='client.admin' cmd=[{"prefix": "mgr module enable", "module": "dashboard"}]: dispatch 2024-01-27T23:25:12.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:11 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:11.238269+0000 mon.smithi121 (mon.0) 87 : audit [INF] from='mgr.14142 172.21.15.121:0/1093011243' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:12.303 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 mgr stat 2024-01-27T23:25:12.719 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.728+0000 7fdeb8474700 1 Processor -- start 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.728+0000 7fdeb8474700 1 -- start start 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.728+0000 7fdeb8474700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb0100da0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.728+0000 7fdeb8474700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb00fc790 con 0x7fdeb0070390 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.728+0000 7fdeb6210700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb0100da0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.728+0000 7fdeb6210700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb0100da0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46376/0 (socket says 172.21.15.121:46376) 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.728+0000 7fdeb6210700 1 -- 172.21.15.121:0/3009509746 learned_addr learned my addr 172.21.15.121:0/3009509746 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.729+0000 7fdeb6210700 1 -- 172.21.15.121:0/3009509746 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeb0101290 con 0x7fdeb0070390 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.729+0000 7fdeb6210700 1 --2- 172.21.15.121:0/3009509746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb0100da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fdeac00b5f0 tx=0x7fdeac006ee0).ready entity=mon.0 client_cookie=a2a8c871635e77b3 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.729+0000 7fdeb520e700 1 -- 172.21.15.121:0/3009509746 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeac00d600 con 0x7fdeb0070390 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.729+0000 7fdeb520e700 1 -- 172.21.15.121:0/3009509746 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeac00dc10 con 0x7fdeb0070390 2024-01-27T23:25:12.720 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.729+0000 7fdeb520e700 1 -- 172.21.15.121:0/3009509746 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeac011cd0 con 0x7fdeb0070390 2024-01-27T23:25:12.722 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.729+0000 7fdeb8474700 1 -- 172.21.15.121:0/3009509746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 msgr2=0x7fdeb0100da0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:12.722 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.729+0000 7fdeb8474700 1 --2- 172.21.15.121:0/3009509746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb0100da0 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fdeac00b5f0 tx=0x7fdeac006ee0).stop 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 -- 172.21.15.121:0/3009509746 shutdown_connections 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 --2- 172.21.15.121:0/3009509746 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb0100da0 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 -- 172.21.15.121:0/3009509746 >> 172.21.15.121:0/3009509746 conn(0x7fdeb00f63c0 msgr2=0x7fdeb00f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 -- 172.21.15.121:0/3009509746 shutdown_connections 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 -- 172.21.15.121:0/3009509746 wait complete. 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 Processor -- start 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 -- start start 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb010ccd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.730+0000 7fdeb8474700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdeb010d1c0 con 0x7fdeb0070390 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fdeb6210700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb010ccd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fdeb6210700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb010ccd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46382/0 (socket says 172.21.15.121:46382) 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fdeb6210700 1 -- 172.21.15.121:0/447017511 learned_addr learned my addr 172.21.15.121:0/447017511 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fdeb6210700 1 -- 172.21.15.121:0/447017511 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdeac00b200 con 0x7fdeb0070390 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fdeb6210700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb010ccd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fdeac00b1d0 tx=0x7fdeac006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:12.723 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fde9effd700 1 -- 172.21.15.121:0/447017511 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeac011df0 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fde9effd700 1 -- 172.21.15.121:0/447017511 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdeac01b8c0 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdeb010d3c0 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fde9effd700 1 -- 172.21.15.121:0/447017511 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdeac01a830 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.731+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdeb010d7d0 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.732+0000 7fde9effd700 1 -- 172.21.15.121:0/447017511 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fdeac00f070 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.732+0000 7fde9effd700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 0x7fde980323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.732+0000 7fde9effd700 1 -- 172.21.15.121:0/447017511 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fdeac01ba30 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.732+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdeb0045ad0 con 0x7fdeb0070390 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.732+0000 7fdeb5a0f700 1 -- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 msgr2=0x7fde980323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1334444756 2024-01-27T23:25:12.724 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.732+0000 7fdeb5a0f700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 0x7fde980323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-27T23:25:12.726 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.737+0000 7fde9effd700 1 -- 172.21.15.121:0/447017511 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdeac016070 con 0x7fdeb0070390 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.930+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mgr stat"} v 0) v1 -- 0x7fdeb0106640 con 0x7fdeb0070390 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.931+0000 7fde9effd700 1 -- 172.21.15.121:0/447017511 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mgr stat"}]=0 v13) v1 ==== 56+0+104 (secure 0 0 0) 0x7fdeac024410 con 0x7fdeb0070390 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "epoch": 13, 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "available": true, 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "active_name": "smithi121.hcppvz", 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "num_standby": 0 2024-01-27T23:25:12.921 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-01-27T23:25:12.922 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb5a0f700 1 -- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 msgr2=0x7fde980323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1334444756 2024-01-27T23:25:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb5a0f700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 0x7fde980323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-27T23:25:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 msgr2=0x7fde980323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T23:25:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 0x7fde980323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:25:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 msgr2=0x7fdeb010ccd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb010ccd0 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7fdeac00b1d0 tx=0x7fdeac006ee0).stop 2024-01-27T23:25:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 shutdown_connections 2024-01-27T23:25:12.923 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fde9802ff00 0x7fde980323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:25:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 --2- 172.21.15.121:0/447017511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdeb0070390 0x7fdeb010ccd0 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 >> 172.21.15.121:0/447017511 conn(0x7fdeb00f63c0 msgr2=0x7fdeb00f7eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 shutdown_connections 2024-01-27T23:25:12.924 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:12.933+0000 7fdeb8474700 1 -- 172.21.15.121:0/447017511 wait complete. 2024-01-27T23:25:13.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:11.934486+0000 mon.smithi121 (mon.0) 88 : audit [INF] from='client.? 172.21.15.121:0/3521685137' entity='client.admin' cmd='[{"prefix": "mgr module enable", "module": "dashboard"}]': finished 2024-01-27T23:25:13.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:11.934690+0000 mon.smithi121 (mon.0) 89 : cluster [DBG] mgrmap e13: smithi121.hcppvz(active, since 7s) 2024-01-27T23:25:13.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:12.932053+0000 mon.smithi121 (mon.0) 90 : audit [DBG] from='client.? 172.21.15.121:0/447017511' entity='client.admin' cmd=[{"prefix": "mgr stat"}]: dispatch 2024-01-27T23:25:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for the mgr to restart... 2024-01-27T23:25:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:Waiting for mgr epoch 13... 2024-01-27T23:25:13.278 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 tell mgr mgr_status 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.621+0000 7fc322fd7700 1 Processor -- start 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.621+0000 7fc322fd7700 1 -- start start 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.622+0000 7fc322fd7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c0fe920 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.622+0000 7fc322fd7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31c0fee10 con 0x7fc31c0703f0 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.622+0000 7fc320d73700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c0fe920 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.622+0000 7fc320d73700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c0fe920 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46386/0 (socket says 172.21.15.121:46386) 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.622+0000 7fc320d73700 1 -- 172.21.15.121:0/2375623421 learned_addr learned my addr 172.21.15.121:0/2375623421 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.622+0000 7fc320d73700 1 -- 172.21.15.121:0/2375623421 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31c0fef50 con 0x7fc31c0703f0 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc320d73700 1 --2- 172.21.15.121:0/2375623421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c0fe920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc31000b5f0 tx=0x7fc310006ee0).ready entity=mon.0 client_cookie=13257669b46fa02 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc31b7fe700 1 -- 172.21.15.121:0/2375623421 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc31000d600 con 0x7fc31c0703f0 2024-01-27T23:25:13.613 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc31b7fe700 1 -- 172.21.15.121:0/2375623421 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc31000dc10 con 0x7fc31c0703f0 2024-01-27T23:25:13.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc31b7fe700 1 -- 172.21.15.121:0/2375623421 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc310011cd0 con 0x7fc31c0703f0 2024-01-27T23:25:13.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc322fd7700 1 -- 172.21.15.121:0/2375623421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 msgr2=0x7fc31c0fe920 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:13.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc322fd7700 1 --2- 172.21.15.121:0/2375623421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c0fe920 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7fc31000b5f0 tx=0x7fc310006ee0).stop 2024-01-27T23:25:13.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc322fd7700 1 -- 172.21.15.121:0/2375623421 shutdown_connections 2024-01-27T23:25:13.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc322fd7700 1 --2- 172.21.15.121:0/2375623421 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c0fe920 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:13.614 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc322fd7700 1 -- 172.21.15.121:0/2375623421 >> 172.21.15.121:0/2375623421 conn(0x7fc31c0f63c0 msgr2=0x7fc31c0f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:13.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc322fd7700 1 -- 172.21.15.121:0/2375623421 shutdown_connections 2024-01-27T23:25:13.615 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.623+0000 7fc322fd7700 1 -- 172.21.15.121:0/2375623421 wait complete. 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.624+0000 7fc322fd7700 1 Processor -- start 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.624+0000 7fc322fd7700 1 -- start start 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.624+0000 7fc322fd7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c10aad0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.624+0000 7fc322fd7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc31c10afc0 con 0x7fc31c0703f0 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.624+0000 7fc320d73700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c10aad0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.624+0000 7fc320d73700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c10aad0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46396/0 (socket says 172.21.15.121:46396) 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.624+0000 7fc320d73700 1 -- 172.21.15.121:0/2372308517 learned_addr learned my addr 172.21.15.121:0/2372308517 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.625+0000 7fc320d73700 1 -- 172.21.15.121:0/2372308517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc31000b200 con 0x7fc31c0703f0 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.625+0000 7fc320d73700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c10aad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc31000df90 tx=0x7fc310006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.625+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc310011da0 con 0x7fc31c0703f0 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.625+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc31001b8c0 con 0x7fc31c0703f0 2024-01-27T23:25:13.616 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.625+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc31001a930 con 0x7fc31c0703f0 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.625+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc31c10b1c0 con 0x7fc31c0703f0 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.625+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc31c10b5d0 con 0x7fc31c0703f0 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.626+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 13) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fc31000f070 con 0x7fc31c0703f0 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.626+0000 7fc3197fa700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.626+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 --> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc300032a80 con 0x7fc30002ff00 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.626+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(3..3 src has 1..3) v4 ==== 1098+0+0 (secure 0 0 0) 0x7fc310041080 con 0x7fc31c0703f0 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.626+0000 7fc31bfff700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 msgr2=0x7fc3000323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1334444756 2024-01-27T23:25:13.617 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.626+0000 7fc31bfff700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.200000 2024-01-27T23:25:13.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.827+0000 7fc31bfff700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 msgr2=0x7fc3000323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1334444756 2024-01-27T23:25:13.816 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:13.827+0000 7fc31bfff700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.400000 2024-01-27T23:25:14.217 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:14.228+0000 7fc31bfff700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 msgr2=0x7fc3000323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1334444756 2024-01-27T23:25:14.217 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:14.228+0000 7fc31bfff700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 0.800000 2024-01-27T23:25:15.018 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:15.029+0000 7fc31bfff700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 msgr2=0x7fc3000323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1334444756 2024-01-27T23:25:15.019 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:15.029+0000 7fc31bfff700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 1.600000 2024-01-27T23:25:16.620 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:16.630+0000 7fc31bfff700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 msgr2=0x7fc3000323c0 unknown :-1 s=STATE_CONNECTING_RE l=1).process reconnect failed to v2:172.21.15.121:6800/1334444756 2024-01-27T23:25:16.621 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:16.630+0000 7fc31bfff700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._fault waiting 3.200000 2024-01-27T23:25:17.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:17.253+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mgrmap(e 14) v1 ==== 35997+0+0 (secure 0 0 0) 0x7fc310010dd0 con 0x7fc31c0703f0 2024-01-27T23:25:17.244 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:17.253+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 msgr2=0x7fc3000323c0 unknown :-1 s=STATE_CONNECTING l=1).mark_down 2024-01-27T23:25:17.244 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:17.253+0000 7fc3197fa700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=START_CONNECT pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:25:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.254+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fc310019ce0 con 0x7fc31c0703f0 2024-01-27T23:25:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.254+0000 7fc3197fa700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc300033b90 0x7fc300035f80 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.254+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 --> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] -- command(tid 0: {"prefix": "get_command_descriptions"}) v1 -- 0x7fc300032a80 con 0x7fc300033b90 2024-01-27T23:25:18.244 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.255+0000 7fc31bfff700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc300033b90 0x7fc300035f80 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.255+0000 7fc31bfff700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc300033b90 0x7fc300035f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc308006fd0 tx=0x7fc308006d70).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:18.245 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.256+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mgr.14162 v2:172.21.15.121:6800/802027800 1 ==== command_reply(tid 0: 0 ) v1 ==== 8+0+6438 (secure 0 0 0) 0x7fc300032a80 con 0x7fc300033b90 2024-01-27T23:25:18.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 --> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] -- command(tid 1: {"prefix": "mgr_status"}) v1 -- 0x7fc31c059100 con 0x7fc300033b90 2024-01-27T23:25:18.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc3197fa700 1 -- 172.21.15.121:0/2372308517 <== mgr.14162 v2:172.21.15.121:6800/802027800 2 ==== command_reply(tid 1: 0 ) v1 ==== 8+0+52 (secure 0 0 0) 0x7fc31c059100 con 0x7fc300033b90 2024-01-27T23:25:18.251 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: { 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "mgrmap_epoch": 15, 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: "initialized": true 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: } 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc300033b90 msgr2=0x7fc300035f80 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc300033b90 0x7fc300035f80 secure :-1 s=READY pgs=2 cs=0 l=1 rev1=1 rx=0x7fc308006fd0 tx=0x7fc308006d70).stop 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 msgr2=0x7fc31c10aad0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c10aad0 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7fc31000df90 tx=0x7fc310006ee0).stop 2024-01-27T23:25:18.253 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 shutdown_connections 2024-01-27T23:25:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc300033b90 0x7fc300035f80 unknown :-1 s=CLOSED pgs=2 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc31c0703f0 0x7fc31c10aad0 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 --2- 172.21.15.121:0/2372308517 >> [v2:172.21.15.121:6800/1334444756,v1:172.21.15.121:6801/1334444756] conn(0x7fc30002ff00 0x7fc3000323c0 unknown :-1 s=CLOSED pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).stop 2024-01-27T23:25:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 >> 172.21.15.121:0/2372308517 conn(0x7fc31c0f63c0 msgr2=0x7fc31c0f7f40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 shutdown_connections 2024-01-27T23:25:18.254 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:18.262+0000 7fc322fd7700 1 -- 172.21.15.121:0/2372308517 wait complete. 2024-01-27T23:25:18.276 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:16.886172+0000 mon.smithi121 (mon.0) 91 : cluster [INF] Active manager daemon smithi121.hcppvz restarted 2024-01-27T23:25:18.276 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:16.886578+0000 mon.smithi121 (mon.0) 92 : cluster [INF] Activating manager daemon smithi121.hcppvz 2024-01-27T23:25:18.276 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:16.888795+0000 mon.smithi121 (mon.0) 93 : cluster [DBG] osdmap e4: 0 total, 0 up, 0 in 2024-01-27T23:25:18.276 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:17.256095+0000 mon.smithi121 (mon.0) 94 : cluster [DBG] mgrmap e14: smithi121.hcppvz(active, starting, since 0.369624s) 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.257213+0000 mon.smithi121 (mon.0) 95 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata", "id": "smithi121"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.257783+0000 mon.smithi121 (mon.0) 96 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mgr metadata", "who": "smithi121.hcppvz", "id": "smithi121.hcppvz"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.258548+0000 mon.smithi121 (mon.0) 97 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mds metadata"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.258777+0000 mon.smithi121 (mon.0) 98 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "osd metadata"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.258996+0000 mon.smithi121 (mon.0) 99 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "mon metadata"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:17.272924+0000 mon.smithi121 (mon.0) 100 : cluster [INF] Manager daemon smithi121.hcppvz is now available 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.371476+0000 mon.smithi121 (mon.0) 101 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.373641+0000 mon.smithi121 (mon.0) 102 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config dump", "format": "json"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.390609+0000 mon.smithi121 (mon.0) 103 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:25:18.277 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:17.398334+0000 mon.smithi121 (mon.0) 104 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:25:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:mgr epoch 13 is available 2024-01-27T23:25:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:Generating a dashboard self-signed certificate... 2024-01-27T23:25:18.594 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 dashboard create-self-signed-cert 2024-01-27T23:25:19.022 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae55247700 1 Processor -- start 2024-01-27T23:25:19.022 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae55247700 1 -- start start 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae55247700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae500fafa0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae55247700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae500fb490 con 0x7fae500fab80 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae4ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae500fafa0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae4ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae500fafa0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46442/0 (socket says 172.21.15.121:46442) 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae4ed9d700 1 -- 172.21.15.121:0/576943435 learned_addr learned my addr 172.21.15.121:0/576943435 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.030+0000 7fae4ed9d700 1 -- 172.21.15.121:0/576943435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae500fb5d0 con 0x7fae500fab80 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae4ed9d700 1 --2- 172.21.15.121:0/576943435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae500fafa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fae4000b5f0 tx=0x7fae40006ee0).ready entity=mon.0 client_cookie=e26d3a17585215a server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae4dd9b700 1 -- 172.21.15.121:0/576943435 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4000d600 con 0x7fae500fab80 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae4dd9b700 1 -- 172.21.15.121:0/576943435 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae4000dc10 con 0x7fae500fab80 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae4dd9b700 1 -- 172.21.15.121:0/576943435 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae40011cd0 con 0x7fae500fab80 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae55247700 1 -- 172.21.15.121:0/576943435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 msgr2=0x7fae500fafa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae55247700 1 --2- 172.21.15.121:0/576943435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae500fafa0 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7fae4000b5f0 tx=0x7fae40006ee0).stop 2024-01-27T23:25:19.023 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae55247700 1 -- 172.21.15.121:0/576943435 shutdown_connections 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae55247700 1 --2- 172.21.15.121:0/576943435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae500fafa0 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae55247700 1 -- 172.21.15.121:0/576943435 >> 172.21.15.121:0/576943435 conn(0x7fae500f63c0 msgr2=0x7fae500f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae55247700 1 -- 172.21.15.121:0/576943435 shutdown_connections 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.031+0000 7fae55247700 1 -- 172.21.15.121:0/576943435 wait complete. 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.032+0000 7fae55247700 1 Processor -- start 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.032+0000 7fae55247700 1 -- start start 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae55247700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae5010d730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae55247700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fae50070470 con 0x7fae500fab80 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae4ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae5010d730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae4ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae5010d730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46452/0 (socket says 172.21.15.121:46452) 2024-01-27T23:25:19.024 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae4ed9d700 1 -- 172.21.15.121:0/2724330019 learned_addr learned my addr 172.21.15.121:0/2724330019 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:19.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae4ed9d700 1 -- 172.21.15.121:0/2724330019 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fae4000b200 con 0x7fae500fab80 2024-01-27T23:25:19.025 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae4ed9d700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae5010d730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fae4000e040 tx=0x7fae40006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4001b410 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fae50070670 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fae4001ba20 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fae4001a910 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.033+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fae5010dde0 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.034+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 15) v1 ==== 36067+0+0 (secure 0 0 0) 0x7fae4000f070 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.034+0000 7fae377fe700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fae3c02fd50 0x7fae3c032210 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.034+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fae50045ad0 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.034+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fae400411b0 con 0x7fae500fab80 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.034+0000 7fae4e59c700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fae3c02fd50 0x7fae3c032210 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:19.026 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.035+0000 7fae4e59c700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fae3c02fd50 0x7fae3c032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fae440096a0 tx=0x7fae44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:19.028 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.038+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fae400107d0 con 0x7fae500fab80 2024-01-27T23:25:19.187 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.197+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 --> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] -- mgr_command(tid 0: {"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}) v1 -- 0x7fae50057f10 con 0x7fae3c02fd50 2024-01-27T23:25:19.245 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.255+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fae40024930 con 0x7fae500fab80 2024-01-27T23:25:19.280 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.291+0000 7fae377fe700 1 -- 172.21.15.121:0/2724330019 <== mgr.14162 v2:172.21.15.121:6800/802027800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+32 (secure 0 0 0) 0x7fae50057f10 con 0x7fae3c02fd50 2024-01-27T23:25:19.281 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: Self-signed certificate created 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.292+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fae3c02fd50 msgr2=0x7fae3c032210 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.292+0000 7fae55247700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fae3c02fd50 0x7fae3c032210 secure :-1 s=READY pgs=6 cs=0 l=1 rev1=1 rx=0x7fae440096a0 tx=0x7fae44006b40).stop 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 msgr2=0x7fae5010d730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae5010d730 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7fae4000e040 tx=0x7fae40006ee0).stop 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 shutdown_connections 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fae3c02fd50 0x7fae3c032210 unknown :-1 s=CLOSED pgs=6 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 --2- 172.21.15.121:0/2724330019 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fae500fab80 0x7fae5010d730 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 >> 172.21.15.121:0/2724330019 conn(0x7fae500f63c0 msgr2=0x7fae500f7e80 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 shutdown_connections 2024-01-27T23:25:19.283 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:19.293+0000 7fae55247700 1 -- 172.21.15.121:0/2724330019 wait complete. 2024-01-27T23:25:19.433 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:19 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:18.151289+0000 mon.smithi121 (mon.0) 105 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:19.434 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:19 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:18.256987+0000 mgr.smithi121.hcppvz (mgr.14162) 1 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "get_command_descriptions"}]: dispatch 2024-01-27T23:25:19.434 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:19 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:18.257782+0000 mon.smithi121 (mon.0) 106 : cluster [DBG] mgrmap e15: smithi121.hcppvz(active, since 1.37131s) 2024-01-27T23:25:19.434 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:19 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:18.263330+0000 mgr.smithi121.hcppvz (mgr.14162) 2 : audit [DBG] from='client.14166 -' entity='client.admin' cmd=[{"prefix": "mgr_status"}]: dispatch 2024-01-27T23:25:19.635 INFO:teuthology.orchestra.run.smithi121.stderr:Creating initial admin user... 2024-01-27T23:25:19.635 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z -v /tmp/ceph-tmph95otsoh:/tmp/dashboard.pw:z docker.io/ceph/ceph:v16.2.4 dashboard ac-user-create admin -i /tmp/dashboard.pw administrator --force-password --pwd-update-required 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.001+0000 7f39db531700 1 Processor -- start 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.001+0000 7f39db531700 1 -- start start 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.001+0000 7f39db531700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d4100d20 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.001+0000 7f39db531700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d40fc710 con 0x7f39d40703f0 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.001+0000 7f39d92cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d4100d20 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.001+0000 7f39d92cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d4100d20 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46458/0 (socket says 172.21.15.121:46458) 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.001+0000 7f39d92cd700 1 -- 172.21.15.121:0/3296084974 learned_addr learned my addr 172.21.15.121:0/3296084974 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.002+0000 7f39d92cd700 1 -- 172.21.15.121:0/3296084974 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39d4101210 con 0x7f39d40703f0 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.002+0000 7f39d92cd700 1 --2- 172.21.15.121:0/3296084974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d4100d20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f39c400b5f0 tx=0x7f39c4006ee0).ready entity=mon.0 client_cookie=38a6271a660a9a91 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.002+0000 7f39c3fff700 1 -- 172.21.15.121:0/3296084974 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39c400d600 con 0x7f39d40703f0 2024-01-27T23:25:19.992 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.002+0000 7f39c3fff700 1 -- 172.21.15.121:0/3296084974 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39c400dc10 con 0x7f39d40703f0 2024-01-27T23:25:19.993 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.002+0000 7f39c3fff700 1 -- 172.21.15.121:0/3296084974 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39c4011cd0 con 0x7f39d40703f0 2024-01-27T23:25:19.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.003+0000 7f39db531700 1 -- 172.21.15.121:0/3296084974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 msgr2=0x7f39d4100d20 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:19.995 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.003+0000 7f39db531700 1 --2- 172.21.15.121:0/3296084974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d4100d20 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f39c400b5f0 tx=0x7f39c4006ee0).stop 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.003+0000 7f39db531700 1 -- 172.21.15.121:0/3296084974 shutdown_connections 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.003+0000 7f39db531700 1 --2- 172.21.15.121:0/3296084974 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d4100d20 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.003+0000 7f39db531700 1 -- 172.21.15.121:0/3296084974 >> 172.21.15.121:0/3296084974 conn(0x7f39d40f63c0 msgr2=0x7f39d40f87e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.003+0000 7f39db531700 1 -- 172.21.15.121:0/3296084974 shutdown_connections 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.003+0000 7f39db531700 1 -- 172.21.15.121:0/3296084974 wait complete. 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39db531700 1 Processor -- start 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39db531700 1 -- start start 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39db531700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d410ccf0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39db531700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f39d410d1e0 con 0x7f39d40703f0 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39d92cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d410ccf0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39d92cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d410ccf0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46470/0 (socket says 172.21.15.121:46470) 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39d92cd700 1 -- 172.21.15.121:0/1538962250 learned_addr learned my addr 172.21.15.121:0/1538962250 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:19.996 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39d92cd700 1 -- 172.21.15.121:0/1538962250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f39c400b200 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39d92cd700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d410ccf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f39c4006a40 tx=0x7f39c4006ee0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39c1ffb700 1 -- 172.21.15.121:0/1538962250 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39c4011c90 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39c1ffb700 1 -- 172.21.15.121:0/1538962250 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f39c401b8c0 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f39d410d3e0 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.004+0000 7f39c1ffb700 1 -- 172.21.15.121:0/1538962250 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f39c401a8c0 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.005+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f39d410d7f0 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.005+0000 7f39c1ffb700 1 -- 172.21.15.121:0/1538962250 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7f39c400f070 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.005+0000 7f39c1ffb700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f39bc02feb0 0x7f39bc032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.006+0000 7f39c1ffb700 1 -- 172.21.15.121:0/1538962250 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f39c4041ab0 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.006+0000 7f39d8acc700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f39bc02feb0 0x7f39bc032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.006+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f39d4045ad0 con 0x7f39d40703f0 2024-01-27T23:25:19.997 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.006+0000 7f39d8acc700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f39bc02feb0 0x7f39bc032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f39d00096a0 tx=0x7f39d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:20.002 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.010+0000 7f39c1ffb700 1 -- 172.21.15.121:0/1538962250 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f39c4016070 con 0x7f39d40703f0 2024-01-27T23:25:20.166 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.177+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 --> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] -- mgr_command(tid 0: {"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}) v1 -- 0x7f39d4058f20 con 0x7f39bc02feb0 2024-01-27T23:25:20.385 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.395+0000 7f39c1ffb700 1 -- 172.21.15.121:0/1538962250 <== mgr.14162 v2:172.21.15.121:6800/802027800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+252 (secure 0 0 0) 0x7f39d4058f20 con 0x7f39bc02feb0 2024-01-27T23:25:20.385 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: {"username": "admin", "password": "$2b$12$V0LJILfEjcpKKYcvBfB3LesBL7rhz/nYqL3.NlTxn/0mxglRbFWhm", "roles": ["administrator"], "name": null, "email": null, "lastUpdate": 1706397920, "enabled": true, "pwdExpirationDate": null, "pwdUpdateRequired": true} 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f39bc02feb0 msgr2=0x7f39bc032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f39bc02feb0 0x7f39bc032370 secure :-1 s=READY pgs=7 cs=0 l=1 rev1=1 rx=0x7f39d00096a0 tx=0x7f39d0006b40).stop 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 msgr2=0x7f39d410ccf0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d410ccf0 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f39c4006a40 tx=0x7f39c4006ee0).stop 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 shutdown_connections 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f39bc02feb0 0x7f39bc032370 unknown :-1 s=CLOSED pgs=7 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 --2- 172.21.15.121:0/1538962250 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f39d40703f0 0x7f39d410ccf0 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.397+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 >> 172.21.15.121:0/1538962250 conn(0x7f39d40f63c0 msgr2=0x7f39d40f7ed0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.398+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 shutdown_connections 2024-01-27T23:25:20.388 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:20.398+0000 7f39db531700 1 -- 172.21.15.121:0/1538962250 wait complete. 2024-01-27T23:25:20.542 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:19.199676+0000 mgr.smithi121.hcppvz (mgr.14162) 3 : audit [DBG] from='client.14172 -' entity='client.admin' cmd=[{"prefix": "dashboard create-self-signed-cert", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:25:20.542 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:19.257959+0000 mon.smithi121 (mon.0) 107 : cluster [DBG] mgrmap e16: smithi121.hcppvz(active, since 2s) 2024-01-27T23:25:20.542 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:19.289730+0000 mon.smithi121 (mon.0) 108 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:20.542 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:19.292286+0000 mon.smithi121 (mon.0) 109 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:20.742 INFO:teuthology.orchestra.run.smithi121.stderr:Fetching dashboard port number... 2024-01-27T23:25:20.742 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint /usr/bin/ceph --init -e CONTAINER_IMAGE=docker.io/ceph/ceph:v16.2.4 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 -v /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715:/var/log/ceph:z -v /tmp/ceph-tmpgblq117f:/etc/ceph/ceph.client.admin.keyring:z -v /tmp/ceph-tmpfyzxr0ou:/etc/ceph/ceph.conf:z docker.io/ceph/ceph:v16.2.4 config get mgr mgr/dashboard/ssl_server_port 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.092+0000 7fbdc4096700 1 Processor -- start 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.092+0000 7fbdc4096700 1 -- start start 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.092+0000 7fbdc4096700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc070ac0 unknown :-1 s=NONE pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.092+0000 7fbdc4096700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc072940 con 0x7fbdbc072520 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.092+0000 7fbdc1e32700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc070ac0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=0 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.092+0000 7fbdc1e32700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc070ac0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=0 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46478/0 (socket says 172.21.15.121:46478) 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.092+0000 7fbdc1e32700 1 -- 172.21.15.121:0/364844184 learned_addr learned my addr 172.21.15.121:0/364844184 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:21.085 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.093+0000 7fbdc1e32700 1 -- 172.21.15.121:0/364844184 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdbc070fb0 con 0x7fbdbc072520 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.093+0000 7fbdc1e32700 1 --2- 172.21.15.121:0/364844184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc070ac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fbdb4009960 tx=0x7fbdb40092d0).ready entity=mon.0 client_cookie=a9ad6b8c4a2c75b4 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.093+0000 7fbdc0e30700 1 -- 172.21.15.121:0/364844184 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb400b9c0 con 0x7fbdbc072520 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.093+0000 7fbdc0e30700 1 -- 172.21.15.121:0/364844184 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdb4011400 con 0x7fbdbc072520 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.093+0000 7fbdc0e30700 1 -- 172.21.15.121:0/364844184 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb4010580 con 0x7fbdbc072520 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 -- 172.21.15.121:0/364844184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 msgr2=0x7fbdbc070ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 --2- 172.21.15.121:0/364844184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc070ac0 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7fbdb4009960 tx=0x7fbdb40092d0).stop 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 -- 172.21.15.121:0/364844184 shutdown_connections 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 --2- 172.21.15.121:0/364844184 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc070ac0 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 -- 172.21.15.121:0/364844184 >> 172.21.15.121:0/364844184 conn(0x7fbdbc0f74b0 msgr2=0x7fbdbc0f98d0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 -- 172.21.15.121:0/364844184 shutdown_connections 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 -- 172.21.15.121:0/364844184 wait complete. 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 Processor -- start 2024-01-27T23:25:21.086 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 -- start start 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc10cd50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.094+0000 7fbdc4096700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbdbc10d240 con 0x7fbdbc072520 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdc1e32700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc10cd50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdc1e32700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc10cd50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:46484/0 (socket says 172.21.15.121:46484) 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdc1e32700 1 -- 172.21.15.121:0/862979376 learned_addr learned my addr 172.21.15.121:0/862979376 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdc1e32700 1 -- 172.21.15.121:0/862979376 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbdb4009570 con 0x7fbdbc072520 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdc1e32700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc10cd50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbdb400ba90 tx=0x7fbdb40092d0).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdba7fc700 1 -- 172.21.15.121:0/862979376 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb40103a0 con 0x7fbdbc072520 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdba7fc700 1 -- 172.21.15.121:0/862979376 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbdb4010eb0 con 0x7fbdbc072520 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbdbc10d440 con 0x7fbdbc072520 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdba7fc700 1 -- 172.21.15.121:0/862979376 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbdb401bc30 con 0x7fbdbc072520 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.095+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbdbc10d850 con 0x7fbdbc072520 2024-01-27T23:25:21.087 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.096+0000 7fbdba7fc700 1 -- 172.21.15.121:0/862979376 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 16) v1 ==== 36208+0+0 (secure 0 0 0) 0x7fbdb400f070 con 0x7fbdbc072520 2024-01-27T23:25:21.088 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.096+0000 7fbdba7fc700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbda802feb0 0x7fbda8032370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:21.088 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.096+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbdbc045ad0 con 0x7fbdbc072520 2024-01-27T23:25:21.088 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.096+0000 7fbdba7fc700 1 -- 172.21.15.121:0/862979376 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbdb4041de0 con 0x7fbdbc072520 2024-01-27T23:25:21.088 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.096+0000 7fbdc1631700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbda802feb0 0x7fbda8032370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:21.088 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.097+0000 7fbdc1631700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbda802feb0 0x7fbda8032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbdb00096a0 tx=0x7fbdb0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:21.089 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.100+0000 7fbdba7fc700 1 -- 172.21.15.121:0/862979376 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbdb4019510 con 0x7fbdbc072520 2024-01-27T23:25:21.240 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.250+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"} v 0) v1 -- 0x7fbdbc101e30 con 0x7fbdbc072520 2024-01-27T23:25:21.240 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.250+0000 7fbdba7fc700 1 -- 172.21.15.121:0/862979376 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]=0 v10) v1 ==== 112+0+5 (secure 0 0 0) 0x7fbdb4019330 con 0x7fbdbc072520 2024-01-27T23:25:21.240 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 8443 2024-01-27T23:25:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbda802feb0 msgr2=0x7fbda8032370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbda802feb0 0x7fbda8032370 secure :-1 s=READY pgs=8 cs=0 l=1 rev1=1 rx=0x7fbdb00096a0 tx=0x7fbdb0006b40).stop 2024-01-27T23:25:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 msgr2=0x7fbdbc10cd50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc10cd50 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7fbdb400ba90 tx=0x7fbdb40092d0).stop 2024-01-27T23:25:21.242 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 shutdown_connections 2024-01-27T23:25:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbda802feb0 0x7fbda8032370 unknown :-1 s=CLOSED pgs=8 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 --2- 172.21.15.121:0/862979376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbdbc072520 0x7fbdbc10cd50 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.252+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 >> 172.21.15.121:0/862979376 conn(0x7fbdbc0f74b0 msgr2=0x7fbdbc0f9050 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.253+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 shutdown_connections 2024-01-27T23:25:21.243 INFO:teuthology.orchestra.run.smithi121.stderr:/usr/bin/ceph: 2024-01-27T23:25:21.253+0000 7fbdc4096700 1 -- 172.21.15.121:0/862979376 wait complete. 2024-01-27T23:25:21.536 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:21 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:20.178911+0000 mgr.smithi121.hcppvz (mgr.14162) 4 : audit [DBG] from='client.14174 -' entity='client.admin' cmd=[{"prefix": "dashboard ac-user-create", "username": "admin", "rolename": "administrator", "force_password": true, "pwd_update_required": true, "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:25:21.536 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:21 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:20.396907+0000 mon.smithi121 (mon.0) 110 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:21.536 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:21 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:21.251606+0000 mon.smithi121 (mon.0) 111 : audit [DBG] from='client.? 172.21.15.121:0/862979376' entity='client.admin' cmd=[{"prefix": "config get", "who": "mgr", "key": "mgr/dashboard/ssl_server_port"}]: dispatch 2024-01-27T23:25:21.592 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-enabled firewalld.service 2024-01-27T23:25:21.600 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: disabled 2024-01-27T23:25:21.601 INFO:teuthology.orchestra.run.smithi121.stderr:Running command: systemctl is-active firewalld.service 2024-01-27T23:25:21.608 INFO:teuthology.orchestra.run.smithi121.stderr:systemctl: inactive 2024-01-27T23:25:21.608 INFO:teuthology.orchestra.run.smithi121.stderr:firewalld.service is not enabled 2024-01-27T23:25:21.609 INFO:teuthology.orchestra.run.smithi121.stderr:Not possible to open ports <[8443]>. firewalld.service is not available 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr:Ceph Dashboard is now available at: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: URL: https://smithi121.front.sepia.ceph.com:8443/ 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: User: admin 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: Password: svo80gz6qz 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr:You can access the Ceph CLI with: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: sudo /home/ubuntu/cephtest/cephadm shell --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr:Please consider enabling telemetry to help improve Ceph: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: ceph telemetry on 2024-01-27T23:25:21.612 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.613 INFO:teuthology.orchestra.run.smithi121.stderr:For more information see: 2024-01-27T23:25:21.613 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.613 INFO:teuthology.orchestra.run.smithi121.stderr: https://docs.ceph.com/docs/pacific/mgr/telemetry/ 2024-01-27T23:25:21.613 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:25:21.613 INFO:teuthology.orchestra.run.smithi121.stderr:Bootstrap complete. 2024-01-27T23:25:21.613 INFO:teuthology.orchestra.run.smithi121.stderr:Releasing lock 139728639085032 on /run/cephadm/2005e414-bd6b-11ee-95b2-87774f69a715.lock 2024-01-27T23:25:21.613 INFO:teuthology.orchestra.run.smithi121.stderr:Lock 139728639085032 released on /run/cephadm/2005e414-bd6b-11ee-95b2-87774f69a715.lock 2024-01-27T23:25:21.646 INFO:tasks.cephadm:Fetching config... 2024-01-27T23:25:21.646 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:25:21.646 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.conf of=/dev/stdout 2024-01-27T23:25:21.665 INFO:tasks.cephadm:Fetching client.admin keyring... 2024-01-27T23:25:21.665 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:25:21.665 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/etc/ceph/ceph.client.admin.keyring of=/dev/stdout 2024-01-27T23:25:21.722 INFO:tasks.cephadm:Fetching mon keyring... 2024-01-27T23:25:21.722 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:25:21.722 DEBUG:teuthology.orchestra.run.smithi121:> sudo dd if=/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/mon.smithi121/keyring of=/dev/stdout 2024-01-27T23:25:21.791 INFO:tasks.cephadm:Fetching pub ssh key... 2024-01-27T23:25:21.791 DEBUG:teuthology.orchestra.run.smithi121:> set -ex 2024-01-27T23:25:21.791 DEBUG:teuthology.orchestra.run.smithi121:> dd if=/home/ubuntu/cephtest/ceph.pub of=/dev/stdout 2024-01-27T23:25:21.849 INFO:tasks.cephadm:Installing pub ssh key for root users... 2024-01-27T23:25:21.849 DEBUG:teuthology.orchestra.run.smithi121:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ2TqBNFQhs8DwLKyxUMf7e4dZd3i+f0IdYVNEgtQjjuetSP+I9E3wInRGjmJqo2JfkJp9zFy/BpByCfdvH5yfJvZ1DIxpkq2mh8MzEpbhxwKczNFTKgzrGbhOi7GthKziXHX2GJ+TW46lDKqz9yrT1Eewv7p7FwvwWk0BDwYObwVZFZpTPugpbUVZq8tC76a3XSjmwTTF+hA+5LlCmd0VMADZbsUg+2zkU6UYapUPPpLxDArJRARIvRSapKz8yiajYUlbbMbf04Yl1hq4j9oPRArWHXACKZaLHDtfy1vCcNmr9vCC6QReEl6vccHq+mH8MAr1wyg3zaF5ZeA2HBUJp0FtgA8S5n1fZdHl0CkCGw4iH43XpwFAqLIbZHOVvglUjqe13Pt6F5VR6RQ/kY31UBDNMRT9qxkJtHBpw3Q0bD2hVfpxdgjekqH4FwhatRsQS8D0R/C+spgd1aYiK3nunG1/tIVLY7RTNRg20LwjiIEPTagwOe1IMdZiAvi5lus= ceph-2005e414-bd6b-11ee-95b2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-27T23:25:21.925 INFO:teuthology.orchestra.run.smithi121.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ2TqBNFQhs8DwLKyxUMf7e4dZd3i+f0IdYVNEgtQjjuetSP+I9E3wInRGjmJqo2JfkJp9zFy/BpByCfdvH5yfJvZ1DIxpkq2mh8MzEpbhxwKczNFTKgzrGbhOi7GthKziXHX2GJ+TW46lDKqz9yrT1Eewv7p7FwvwWk0BDwYObwVZFZpTPugpbUVZq8tC76a3XSjmwTTF+hA+5LlCmd0VMADZbsUg+2zkU6UYapUPPpLxDArJRARIvRSapKz8yiajYUlbbMbf04Yl1hq4j9oPRArWHXACKZaLHDtfy1vCcNmr9vCC6QReEl6vccHq+mH8MAr1wyg3zaF5ZeA2HBUJp0FtgA8S5n1fZdHl0CkCGw4iH43XpwFAqLIbZHOVvglUjqe13Pt6F5VR6RQ/kY31UBDNMRT9qxkJtHBpw3Q0bD2hVfpxdgjekqH4FwhatRsQS8D0R/C+spgd1aYiK3nunG1/tIVLY7RTNRg20LwjiIEPTagwOe1IMdZiAvi5lus= ceph-2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:25:21.936 DEBUG:teuthology.orchestra.run.smithi194:> sudo install -d -m 0700 /root/.ssh && echo 'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ2TqBNFQhs8DwLKyxUMf7e4dZd3i+f0IdYVNEgtQjjuetSP+I9E3wInRGjmJqo2JfkJp9zFy/BpByCfdvH5yfJvZ1DIxpkq2mh8MzEpbhxwKczNFTKgzrGbhOi7GthKziXHX2GJ+TW46lDKqz9yrT1Eewv7p7FwvwWk0BDwYObwVZFZpTPugpbUVZq8tC76a3XSjmwTTF+hA+5LlCmd0VMADZbsUg+2zkU6UYapUPPpLxDArJRARIvRSapKz8yiajYUlbbMbf04Yl1hq4j9oPRArWHXACKZaLHDtfy1vCcNmr9vCC6QReEl6vccHq+mH8MAr1wyg3zaF5ZeA2HBUJp0FtgA8S5n1fZdHl0CkCGw4iH43XpwFAqLIbZHOVvglUjqe13Pt6F5VR6RQ/kY31UBDNMRT9qxkJtHBpw3Q0bD2hVfpxdgjekqH4FwhatRsQS8D0R/C+spgd1aYiK3nunG1/tIVLY7RTNRg20LwjiIEPTagwOe1IMdZiAvi5lus= ceph-2005e414-bd6b-11ee-95b2-87774f69a715' | sudo tee -a /root/.ssh/authorized_keys && sudo chmod 0600 /root/.ssh/authorized_keys 2024-01-27T23:25:21.980 INFO:teuthology.orchestra.run.smithi194.stdout:ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQDJ2TqBNFQhs8DwLKyxUMf7e4dZd3i+f0IdYVNEgtQjjuetSP+I9E3wInRGjmJqo2JfkJp9zFy/BpByCfdvH5yfJvZ1DIxpkq2mh8MzEpbhxwKczNFTKgzrGbhOi7GthKziXHX2GJ+TW46lDKqz9yrT1Eewv7p7FwvwWk0BDwYObwVZFZpTPugpbUVZq8tC76a3XSjmwTTF+hA+5LlCmd0VMADZbsUg+2zkU6UYapUPPpLxDArJRARIvRSapKz8yiajYUlbbMbf04Yl1hq4j9oPRArWHXACKZaLHDtfy1vCcNmr9vCC6QReEl6vccHq+mH8MAr1wyg3zaF5ZeA2HBUJp0FtgA8S5n1fZdHl0CkCGw4iH43XpwFAqLIbZHOVvglUjqe13Pt6F5VR6RQ/kY31UBDNMRT9qxkJtHBpw3Q0bD2hVfpxdgjekqH4FwhatRsQS8D0R/C+spgd1aYiK3nunG1/tIVLY7RTNRg20LwjiIEPTagwOe1IMdZiAvi5lus= ceph-2005e414-bd6b-11ee-95b2-87774f69a715 2024-01-27T23:25:21.992 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph config set mgr mgr/cephadm/allow_ptrace true 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.291+0000 7fdb1b59e700 1 -- 172.21.15.121:0/2751163941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 msgr2=0x7fdb1c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.291+0000 7fdb1b59e700 1 --2- 172.21.15.121:0/2751163941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f55c0 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7fdb10009fb0 tx=0x7fdb10009b30).stop 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.291+0000 7fdb1b59e700 1 -- 172.21.15.121:0/2751163941 shutdown_connections 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.291+0000 7fdb1b59e700 1 --2- 172.21.15.121:0/2751163941 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0f55c0 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.291+0000 7fdb1b59e700 1 -- 172.21.15.121:0/2751163941 >> 172.21.15.121:0/2751163941 conn(0x7fdb1c0f09a0 msgr2=0x7fdb1c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.291+0000 7fdb1b59e700 1 -- 172.21.15.121:0/2751163941 shutdown_connections 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.291+0000 7fdb1b59e700 1 -- 172.21.15.121:0/2751163941 wait complete. 2024-01-27T23:25:23.282 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1b59e700 1 Processor -- start 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1b59e700 1 -- start start 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1b59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1b59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdb1c0fb5a0 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:52140/0 (socket says 172.21.15.121:52140) 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1a59c700 1 -- 172.21.15.121:0/1110314451 learned_addr learned my addr 172.21.15.121:0/1110314451 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1a59c700 1 -- 172.21.15.121:0/1110314451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdb10005040 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb1a59c700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0fd120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fdb10013510 tx=0x7fdb1000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.292+0000 7fdb0affd700 1 -- 172.21.15.121:0/1110314451 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb10009320 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.283 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.293+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdb1c0fb740 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.293+0000 7fdb0affd700 1 -- 172.21.15.121:0/1110314451 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdb10013f00 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.293+0000 7fdb0affd700 1 -- 172.21.15.121:0/1110314451 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdb10014dc0 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.293+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdb1c0fbbd0 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.293+0000 7fdb0affd700 1 -- 172.21.15.121:0/1110314451 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdb1001d3d0 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.294+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdb1c059100 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.294+0000 7fdb0affd700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdb0402ff60 0x7fdb04032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:23.284 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.294+0000 7fdb0affd700 1 -- 172.21.15.121:0/1110314451 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdb10041040 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.294+0000 7fdb19d9b700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdb0402ff60 0x7fdb04032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:23.285 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.294+0000 7fdb19d9b700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdb0402ff60 0x7fdb04032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fdb0c0096a0 tx=0x7fdb0c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:23.287 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.298+0000 7fdb0affd700 1 -- 172.21.15.121:0/1110314451 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdb100126f0 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.450 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.461+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command([{prefix=config set, name=mgr/cephadm/allow_ptrace}] v 0) v1 -- 0x7fdb1c0fabe0 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.453 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.464+0000 7fdb0affd700 1 -- 172.21.15.121:0/1110314451 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{prefix=config set, name=mgr/cephadm/allow_ptrace}]=0 v11)=0 v11) v1 ==== 125+0+0 (secure 0 0 0) 0x7fdb10012330 con 0x7fdb1c0f51a0 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdb0402ff60 msgr2=0x7fdb04032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdb0402ff60 0x7fdb04032420 secure :-1 s=READY pgs=9 cs=0 l=1 rev1=1 rx=0x7fdb0c0096a0 tx=0x7fdb0c006b40).stop 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 msgr2=0x7fdb1c0fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0fd120 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fdb10013510 tx=0x7fdb1000b040).stop 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 shutdown_connections 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdb0402ff60 0x7fdb04032420 unknown :-1 s=CLOSED pgs=9 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 --2- 172.21.15.121:0/1110314451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdb1c0f51a0 0x7fdb1c0fd120 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 >> 172.21.15.121:0/1110314451 conn(0x7fdb1c0f09a0 msgr2=0x7fdb1c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:23.459 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 shutdown_connections 2024-01-27T23:25:23.460 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:23.466+0000 7fdb1b59e700 1 -- 172.21.15.121:0/1110314451 wait complete. 2024-01-27T23:25:23.829 INFO:tasks.cephadm:Writing (initial) conf and keyring to smithi194 2024-01-27T23:25:23.829 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:25:23.829 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.conf 2024-01-27T23:25:23.852 DEBUG:teuthology.orchestra.run.smithi194:> set -ex 2024-01-27T23:25:23.852 DEBUG:teuthology.orchestra.run.smithi194:> dd of=/etc/ceph/ceph.client.admin.keyring 2024-01-27T23:25:23.913 INFO:tasks.cephadm:Adding host smithi194 to orchestrator... 2024-01-27T23:25:23.913 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph orch host add smithi194 2024-01-27T23:25:24.324 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:23 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:22.952535+0000 mon.smithi121 (mon.0) 112 : cluster [DBG] mgrmap e17: smithi121.hcppvz(active, since 6s) 2024-01-27T23:25:24.324 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:23 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:23.465215+0000 mon.smithi121 (mon.0) 113 : audit [INF] from='client.? 172.21.15.121:0/1110314451' entity='client.admin' 2024-01-27T23:25:26.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:24.707852+0000 mon.smithi121 (mon.0) 114 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:26.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:24.987994+0000 mon.smithi121 (mon.0) 115 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:27.086 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:25:24.994141+0000 mgr.smithi121.hcppvz (mgr.14162) 5 : cephadm [INF] Deploying daemon alertmanager.smithi121 on smithi121 2024-01-27T23:25:38.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:37.259727+0000 mgr.smithi121.hcppvz (mgr.14162) 6 : cluster [DBG] pgmap v3: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:40.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:39.260031+0000 mgr.smithi121.hcppvz (mgr.14162) 7 : cluster [DBG] pgmap v4: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:42.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:41.260268+0000 mgr.smithi121.hcppvz (mgr.14162) 8 : cluster [DBG] pgmap v5: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:44.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:43.260450+0000 mgr.smithi121.hcppvz (mgr.14162) 9 : cluster [DBG] pgmap v6: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.333+0000 7f34ad4f0700 1 -- 172.21.15.194:0/2929566811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 msgr2=0x7f34a80f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.333+0000 7f34ad4f0700 1 --2- 172.21.15.194:0/2929566811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a80f66c0 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f3498009fb0 tx=0x7f3498009b30).stop 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.333+0000 7f34ad4f0700 1 -- 172.21.15.194:0/2929566811 shutdown_connections 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.333+0000 7f34ad4f0700 1 --2- 172.21.15.194:0/2929566811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a80f66c0 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.333+0000 7f34ad4f0700 1 -- 172.21.15.194:0/2929566811 >> 172.21.15.194:0/2929566811 conn(0x7f34a80f1aa0 msgr2=0x7f34a80f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.333+0000 7f34ad4f0700 1 -- 172.21.15.194:0/2929566811 shutdown_connections 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.333+0000 7f34ad4f0700 1 -- 172.21.15.194:0/2929566811 wait complete. 2024-01-27T23:25:46.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.334+0000 7f34ad4f0700 1 Processor -- start 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.334+0000 7f34ad4f0700 1 -- start start 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.334+0000 7f34ad4f0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a81064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.334+0000 7f34ad4f0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34a81069a0 con 0x7f34a80f62a0 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a81064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a81064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:50872/0 (socket says 172.21.15.194:50872) 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a7fff700 1 -- 172.21.15.194:0/236250148 learned_addr learned my addr 172.21.15.194:0/236250148 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a7fff700 1 -- 172.21.15.194:0/236250148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3498005040 con 0x7f34a80f62a0 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a7fff700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a81064b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3498013510 tx=0x7f349800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a4ff9700 1 -- 172.21.15.194:0/236250148 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3498009320 con 0x7f34a80f62a0 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34a8106ba0 con 0x7f34a80f62a0 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a4ff9700 1 -- 172.21.15.194:0/236250148 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3498013f00 con 0x7f34a80f62a0 2024-01-27T23:25:46.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34a4ff9700 1 -- 172.21.15.194:0/236250148 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3498014dc0 con 0x7f34a80f62a0 2024-01-27T23:25:46.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.335+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34a8107030 con 0x7f34a80f62a0 2024-01-27T23:25:46.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.336+0000 7f34a4ff9700 1 -- 172.21.15.194:0/236250148 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3498014420 con 0x7f34a80f62a0 2024-01-27T23:25:46.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.336+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34a8045ad0 con 0x7f34a80f62a0 2024-01-27T23:25:46.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.336+0000 7f34a4ff9700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:46.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.336+0000 7f34a4ff9700 1 -- 172.21.15.194:0/236250148 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3498041020 con 0x7f34a80f62a0 2024-01-27T23:25:46.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.336+0000 7f34a77fe700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:46.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.337+0000 7f34a77fe700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f349002ff60 0x7f3490032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f349c0096a0 tx=0x7f349c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:46.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.341+0000 7f34a4ff9700 1 -- 172.21.15.194:0/236250148 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34980095f0 con 0x7f34a80f62a0 2024-01-27T23:25:46.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:46.510+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 --> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] -- mgr_command(tid 0: {"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}) v1 -- 0x7f34a80f2c20 con 0x7f349002ff60 2024-01-27T23:25:46.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:45.260631+0000 mgr.smithi121.hcppvz (mgr.14162) 10 : cluster [DBG] pgmap v7: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:47.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.112+0000 7f34a4ff9700 1 -- 172.21.15.194:0/236250148 <== mgr.14162 v2:172.21.15.121:6800/802027800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+23 (secure 0 0 0) 0x7f34a80f2c20 con 0x7f349002ff60 2024-01-27T23:25:47.102 INFO:teuthology.orchestra.run.smithi194.stdout:Added host 'smithi194' 2024-01-27T23:25:47.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f349002ff60 msgr2=0x7f3490032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:47.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f349002ff60 0x7f3490032420 secure :-1 s=READY pgs=10 cs=0 l=1 rev1=1 rx=0x7f349c0096a0 tx=0x7f349c006b40).stop 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 msgr2=0x7f34a81064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a81064b0 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f3498013510 tx=0x7f349800b040).stop 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 shutdown_connections 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f349002ff60 0x7f3490032420 unknown :-1 s=CLOSED pgs=10 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 --2- 172.21.15.194:0/236250148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34a80f62a0 0x7f34a81064b0 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 >> 172.21.15.194:0/236250148 conn(0x7f34a80f1aa0 msgr2=0x7f34a80f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 shutdown_connections 2024-01-27T23:25:47.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:47.114+0000 7f34ad4f0700 1 -- 172.21.15.194:0/236250148 wait complete. 2024-01-27T23:25:47.438 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph orch host ls --format=json 2024-01-27T23:25:47.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:47 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:46.512091+0000 mgr.smithi121.hcppvz (mgr.14162) 11 : audit [DBG] from='client.14180 -' entity='client.admin' cmd=[{"prefix": "orch host add", "hostname": "smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:25:47.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:47 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:47.112361+0000 mon.smithi121 (mon.0) 116 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:48.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:25:47.112789+0000 mgr.smithi121.hcppvz (mgr.14162) 12 : cephadm [INF] Added host smithi194 2024-01-27T23:25:48.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:47.260863+0000 mgr.smithi121.hcppvz (mgr.14162) 13 : cluster [DBG] pgmap v8: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:48.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.626+0000 7f4ca6f44700 1 -- 172.21.15.194:0/4174958646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 msgr2=0x7f4ca00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:48.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.626+0000 7f4ca6f44700 1 --2- 172.21.15.194:0/4174958646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca00f55c0 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7f4c90009fb0 tx=0x7f4c90009b30).stop 2024-01-27T23:25:48.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 -- 172.21.15.194:0/4174958646 shutdown_connections 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 --2- 172.21.15.194:0/4174958646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca00f55c0 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 -- 172.21.15.194:0/4174958646 >> 172.21.15.194:0/4174958646 conn(0x7f4ca00f09a0 msgr2=0x7f4ca00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 -- 172.21.15.194:0/4174958646 shutdown_connections 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 -- 172.21.15.194:0/4174958646 wait complete. 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 Processor -- start 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 -- start start 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca01064c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca6f44700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ca01069b0 con 0x7f4ca00f51a0 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca5f42700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca01064c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca5f42700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca01064c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:50896/0 (socket says 172.21.15.194:50896) 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.627+0000 7f4ca5f42700 1 -- 172.21.15.194:0/1802486929 learned_addr learned my addr 172.21.15.194:0/1802486929 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.628+0000 7f4ca5f42700 1 -- 172.21.15.194:0/1802486929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4c90005040 con 0x7f4ca00f51a0 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.628+0000 7f4ca5f42700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca01064c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f4c90013510 tx=0x7f4c9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:48.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.628+0000 7f4c967fc700 1 -- 172.21.15.194:0/1802486929 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c90009320 con 0x7f4ca00f51a0 2024-01-27T23:25:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.628+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ca0106bb0 con 0x7f4ca00f51a0 2024-01-27T23:25:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.628+0000 7f4c967fc700 1 -- 172.21.15.194:0/1802486929 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4c90013f00 con 0x7f4ca00f51a0 2024-01-27T23:25:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.628+0000 7f4c967fc700 1 -- 172.21.15.194:0/1802486929 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4c90014dc0 con 0x7f4ca00f51a0 2024-01-27T23:25:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.628+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ca0107040 con 0x7f4ca00f51a0 2024-01-27T23:25:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.629+0000 7f4c967fc700 1 -- 172.21.15.194:0/1802486929 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4c900095f0 con 0x7f4ca00f51a0 2024-01-27T23:25:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.629+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ca0045ad0 con 0x7f4ca00f51a0 2024-01-27T23:25:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.629+0000 7f4c967fc700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4c8c02ffb0 0x7f4c8c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.629+0000 7f4c967fc700 1 -- 172.21.15.194:0/1802486929 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4c900410e0 con 0x7f4ca00f51a0 2024-01-27T23:25:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.629+0000 7f4ca5741700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4c8c02ffb0 0x7f4c8c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.630+0000 7f4ca5741700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4c8c02ffb0 0x7f4c8c032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4c9c0096a0 tx=0x7f4c9c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:48.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.634+0000 7f4c967fc700 1 -- 172.21.15.194:0/1802486929 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4c90014420 con 0x7f4ca00f51a0 2024-01-27T23:25:48.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.803+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 --> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] -- mgr_command(tid 0: {"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}) v1 -- 0x7f4ca00f1b70 con 0x7f4c8c02ffb0 2024-01-27T23:25:48.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.805+0000 7f4c967fc700 1 -- 172.21.15.194:0/1802486929 <== mgr.14162 v2:172.21.15.121:6800/802027800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+157 (secure 0 0 0) 0x7f4ca00f1b70 con 0x7f4c8c02ffb0 2024-01-27T23:25:48.794 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:25:48.794 INFO:teuthology.orchestra.run.smithi194.stdout:[{"addr": "172.21.15.121", "hostname": "smithi121", "labels": [], "status": ""}, {"addr": "smithi194", "hostname": "smithi194", "labels": [], "status": ""}] 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4c8c02ffb0 msgr2=0x7f4c8c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4c8c02ffb0 0x7f4c8c032470 secure :-1 s=READY pgs=11 cs=0 l=1 rev1=1 rx=0x7f4c9c0096a0 tx=0x7f4c9c006b40).stop 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 msgr2=0x7f4ca01064c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca01064c0 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f4c90013510 tx=0x7f4c9000b040).stop 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 shutdown_connections 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4c8c02ffb0 0x7f4c8c032470 unknown :-1 s=CLOSED pgs=11 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 --2- 172.21.15.194:0/1802486929 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ca00f51a0 0x7f4ca01064c0 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 >> 172.21.15.194:0/1802486929 conn(0x7f4ca00f09a0 msgr2=0x7f4ca00f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:48.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 shutdown_connections 2024-01-27T23:25:48.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:48.806+0000 7f4ca6f44700 1 -- 172.21.15.194:0/1802486929 wait complete. 2024-01-27T23:25:49.175 INFO:tasks.cephadm:Setting crush tunables to default 2024-01-27T23:25:49.175 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph osd crush tunables default 2024-01-27T23:25:49.493 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:49 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:48.805747+0000 mgr.smithi121.hcppvz (mgr.14162) 14 : audit [DBG] from='client.14182 -' entity='client.admin' cmd=[{"prefix": "orch host ls", "target": ["mon-mgr", ""], "format": "json"}]: dispatch 2024-01-27T23:25:50.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.330+0000 7f9ba1265700 1 -- 172.21.15.121:0/1328543617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 msgr2=0x7f9b9c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:50.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.330+0000 7f9ba1265700 1 --2- 172.21.15.121:0/1328543617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c0f5830 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f9b90009fb0 tx=0x7f9b90009b30).stop 2024-01-27T23:25:50.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.330+0000 7f9ba1265700 1 -- 172.21.15.121:0/1328543617 shutdown_connections 2024-01-27T23:25:50.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.330+0000 7f9ba1265700 1 --2- 172.21.15.121:0/1328543617 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c0f5830 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:50.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.330+0000 7f9ba1265700 1 -- 172.21.15.121:0/1328543617 >> 172.21.15.121:0/1328543617 conn(0x7f9b9c0f0b70 msgr2=0x7f9b9c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:50.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.330+0000 7f9ba1265700 1 -- 172.21.15.121:0/1328543617 shutdown_connections 2024-01-27T23:25:50.319 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.330+0000 7f9ba1265700 1 -- 172.21.15.121:0/1328543617 wait complete. 2024-01-27T23:25:50.321 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.331+0000 7f9ba1265700 1 Processor -- start 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.331+0000 7f9ba1265700 1 -- start start 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.331+0000 7f9ba1265700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.331+0000 7f9ba1265700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b9c060240 con 0x7f9b9c0f5410 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.331+0000 7f9b9affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.331+0000 7f9b9affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.121:39124/0 (socket says 172.21.15.121:39124) 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.331+0000 7f9b9affd700 1 -- 172.21.15.121:0/4285117257 learned_addr learned my addr 172.21.15.121:0/4285117257 (peer_addr_for_me v2:172.21.15.121:0/0) 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.332+0000 7f9b9affd700 1 -- 172.21.15.121:0/4285117257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b90005040 con 0x7f9b9c0f5410 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.332+0000 7f9b9affd700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c109430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f9b90013510 tx=0x7f9b9000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.332+0000 7f9b837fe700 1 -- 172.21.15.121:0/4285117257 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b90009320 con 0x7f9b9c0f5410 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.332+0000 7f9b837fe700 1 -- 172.21.15.121:0/4285117257 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b90013f00 con 0x7f9b9c0f5410 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.332+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b9c060440 con 0x7f9b9c0f5410 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.332+0000 7f9b837fe700 1 -- 172.21.15.121:0/4285117257 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b90014d90 con 0x7f9b9c0f5410 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.332+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b9c109ae0 con 0x7f9b9c0f5410 2024-01-27T23:25:50.322 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.333+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b9c045ad0 con 0x7f9b9c0f5410 2024-01-27T23:25:50.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.333+0000 7f9b837fe700 1 -- 172.21.15.121:0/4285117257 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b900095f0 con 0x7f9b9c0f5410 2024-01-27T23:25:50.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.333+0000 7f9b837fe700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b8402ff60 0x7f9b84032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:50.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.333+0000 7f9b837fe700 1 -- 172.21.15.121:0/4285117257 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(4..4 src has 1..4) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b90041000 con 0x7f9b9c0f5410 2024-01-27T23:25:50.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.333+0000 7f9b9a7fc700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b8402ff60 0x7f9b84032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:50.323 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.334+0000 7f9b9a7fc700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b8402ff60 0x7f9b84032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f9b8c0096a0 tx=0x7f9b8c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:50.325 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.337+0000 7f9b837fe700 1 -- 172.21.15.121:0/4285117257 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b90014420 con 0x7f9b9c0f5410 2024-01-27T23:25:50.478 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:50.490+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "osd crush tunables", "profile": "default"} v 0) v1 -- 0x7f9b9c0fae60 con 0x7f9b9c0f5410 2024-01-27T23:25:50.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:49.261150+0000 mgr.smithi121.hcppvz (mgr.14162) 15 : cluster [DBG] pgmap v9: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:51.310 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.322+0000 7f9b837fe700 1 -- 172.21.15.121:0/4285117257 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "osd crush tunables", "profile": "default"}]=0 adjusted tunables profile to default v5) v1 ==== 124+0+0 (secure 0 0 0) 0x7f9b9001d7f0 con 0x7f9b9c0f5410 2024-01-27T23:25:51.312 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b8402ff60 msgr2=0x7f9b84032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b8402ff60 0x7f9b84032420 secure :-1 s=READY pgs=12 cs=0 l=1 rev1=1 rx=0x7f9b8c0096a0 tx=0x7f9b8c006b40).stop 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 msgr2=0x7f9b9c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c109430 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f9b90013510 tx=0x7f9b9000b040).stop 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 shutdown_connections 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b8402ff60 0x7f9b84032420 unknown :-1 s=CLOSED pgs=12 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 --2- 172.21.15.121:0/4285117257 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b9c0f5410 0x7f9b9c109430 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 >> 172.21.15.121:0/4285117257 conn(0x7f9b9c0f0b70 msgr2=0x7f9b9c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 shutdown_connections 2024-01-27T23:25:51.313 INFO:teuthology.orchestra.run.smithi121.stderr:2024-01-27T23:25:51.324+0000 7f9ba1265700 1 -- 172.21.15.121:0/4285117257 wait complete. 2024-01-27T23:25:51.314 INFO:teuthology.orchestra.run.smithi121.stderr:adjusted tunables profile to default 2024-01-27T23:25:51.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:51 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:50.490977+0000 mon.smithi121 (mon.0) 117 : audit [INF] from='client.? 172.21.15.121:0/4285117257' entity='client.admin' cmd=[{"prefix": "osd crush tunables", "profile": "default"}]: dispatch 2024-01-27T23:25:51.647 INFO:tasks.cephadm:Adding mon.smithi121 on smithi121 2024-01-27T23:25:51.647 INFO:tasks.cephadm:Adding mon.smithi194 on smithi194 2024-01-27T23:25:51.647 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph orch apply mon '2;smithi121:172.21.15.121=smithi121;smithi194:172.21.15.194=smithi194' 2024-01-27T23:25:52.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:51.261625+0000 mgr.smithi121.hcppvz (mgr.14162) 16 : cluster [DBG] pgmap v10: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:52.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:51.322468+0000 mon.smithi121 (mon.0) 118 : audit [INF] from='client.? 172.21.15.121:0/4285117257' entity='client.admin' cmd='[{"prefix": "osd crush tunables", "profile": "default"}]': finished 2024-01-27T23:25:52.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:51.322513+0000 mon.smithi121 (mon.0) 119 : cluster [DBG] osdmap e5: 0 total, 0 up, 0 in 2024-01-27T23:25:52.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.818+0000 7f72fffa4700 1 -- 172.21.15.194:0/2149559688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 msgr2=0x7f72f80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:52.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.818+0000 7f72fffa4700 1 --2- 172.21.15.194:0/2149559688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f80f6930 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f72e8009fb0 tx=0x7f72e8009b30).stop 2024-01-27T23:25:52.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.818+0000 7f72fffa4700 1 -- 172.21.15.194:0/2149559688 shutdown_connections 2024-01-27T23:25:52.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.818+0000 7f72fffa4700 1 --2- 172.21.15.194:0/2149559688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f80f6930 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:52.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.818+0000 7f72fffa4700 1 -- 172.21.15.194:0/2149559688 >> 172.21.15.194:0/2149559688 conn(0x7f72f80f1c70 msgr2=0x7f72f80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.818+0000 7f72fffa4700 1 -- 172.21.15.194:0/2149559688 shutdown_connections 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.818+0000 7f72fffa4700 1 -- 172.21.15.194:0/2149559688 wait complete. 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fffa4700 1 Processor -- start 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fffa4700 1 -- start start 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fffa4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fffa4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f72f8106c20 con 0x7f72f80f6510 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fdd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fdd40700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:50926/0 (socket says 172.21.15.194:50926) 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fdd40700 1 -- 172.21.15.194:0/632681479 learned_addr learned my addr 172.21.15.194:0/632681479 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.819+0000 7f72fdd40700 1 -- 172.21.15.194:0/632681479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f72e8005040 con 0x7f72f80f6510 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.820+0000 7f72fdd40700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f8106730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f72e8013510 tx=0x7f72e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.820+0000 7f72ee7fc700 1 -- 172.21.15.194:0/632681479 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72e8009320 con 0x7f72f80f6510 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.820+0000 7f72ee7fc700 1 -- 172.21.15.194:0/632681479 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f72e8013f00 con 0x7f72f80f6510 2024-01-27T23:25:52.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.820+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f72f8106e20 con 0x7f72f80f6510 2024-01-27T23:25:52.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.820+0000 7f72ee7fc700 1 -- 172.21.15.194:0/632681479 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f72e8014d90 con 0x7f72f80f6510 2024-01-27T23:25:52.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.820+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f72f81072b0 con 0x7f72f80f6510 2024-01-27T23:25:52.812 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.820+0000 7f72ee7fc700 1 -- 172.21.15.194:0/632681479 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f72e80263a0 con 0x7f72f80f6510 2024-01-27T23:25:52.812 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.821+0000 7f72ee7fc700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f72e402ff60 0x7f72e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:52.812 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.821+0000 7f72ee7fc700 1 -- 172.21.15.194:0/632681479 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f72e80410e0 con 0x7f72f80f6510 2024-01-27T23:25:52.812 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.821+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f72f8045ad0 con 0x7f72f80f6510 2024-01-27T23:25:52.812 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.821+0000 7f72fd53f700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f72e402ff60 0x7f72e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:52.812 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.821+0000 7f72fd53f700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f72e402ff60 0x7f72e4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f72f40096a0 tx=0x7f72f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:52.814 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:52.825+0000 7f72ee7fc700 1 -- 172.21.15.194:0/632681479 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f72e8012920 con 0x7f72f80f6510 2024-01-27T23:25:52.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.002+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 --> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] -- mgr_command(tid 0: {"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi121:172.21.15.121=smithi121;smithi194:172.21.15.194=smithi194", "target": ["mon-mgr", ""]}) v1 -- 0x7f72f80f2f20 con 0x7f72e402ff60 2024-01-27T23:25:52.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.007+0000 7f72ee7fc700 1 -- 172.21.15.194:0/632681479 <== mgr.14162 v2:172.21.15.121:6800/802027800 1 ==== mgr_command_reply(tid 0: 0 ) v1 ==== 8+0+24 (secure 0 0 0) 0x7f72f80f2f20 con 0x7f72e402ff60 2024-01-27T23:25:52.997 INFO:teuthology.orchestra.run.smithi194.stdout:Scheduled mon update... 2024-01-27T23:25:52.998 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f72e402ff60 msgr2=0x7f72e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f72e402ff60 0x7f72e4032420 secure :-1 s=READY pgs=13 cs=0 l=1 rev1=1 rx=0x7f72f40096a0 tx=0x7f72f4006b40).stop 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 msgr2=0x7f72f8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f8106730 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f72e8013510 tx=0x7f72e800b040).stop 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 shutdown_connections 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f72e402ff60 0x7f72e4032420 unknown :-1 s=CLOSED pgs=13 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 --2- 172.21.15.194:0/632681479 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f72f80f6510 0x7f72f8106730 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 >> 172.21.15.194:0/632681479 conn(0x7f72f80f1c70 msgr2=0x7f72f80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 shutdown_connections 2024-01-27T23:25:52.999 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:53.009+0000 7f72fffa4700 1 -- 172.21.15.194:0/632681479 wait complete. 2024-01-27T23:25:53.364 DEBUG:teuthology.orchestra.run.smithi194:mon.smithi194> sudo journalctl -f -n 0 -u ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194.service 2024-01-27T23:25:53.367 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:25:53.367 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:25:53.395 INFO:journalctl@ceph.mon.smithi194.smithi194.stdout:-- Logs begin at Sat 2024-01-27 23:13:56 UTC. -- 2024-01-27T23:25:54.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:53.004257+0000 mgr.smithi121.hcppvz (mgr.14162) 17 : audit [DBG] from='client.14186 -' entity='client.admin' cmd=[{"prefix": "orch apply", "service_type": "mon", "placement": "2;smithi121:172.21.15.121=smithi121;smithi194:172.21.15.194=smithi194", "target": ["mon-mgr", ""]}]: dispatch 2024-01-27T23:25:54.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:25:53.005614+0000 mgr.smithi121.hcppvz (mgr.14162) 18 : cephadm [INF] Saving service mon spec with placement smithi121:172.21.15.121=smithi121;smithi194:172.21.15.194=smithi194;count:2 2024-01-27T23:25:54.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:53.008155+0000 mon.smithi121 (mon.0) 120 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:25:54.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:53.261875+0000 mgr.smithi121.hcppvz (mgr.14162) 19 : cluster [DBG] pgmap v12: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:54.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.520+0000 7fd047f79700 1 -- 172.21.15.194:0/2871756934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 msgr2=0x7fd0400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:54.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.520+0000 7fd047f79700 1 --2- 172.21.15.194:0/2871756934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0400f5830 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7fd034009fb0 tx=0x7fd034009b30).stop 2024-01-27T23:25:54.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.520+0000 7fd047f79700 1 -- 172.21.15.194:0/2871756934 shutdown_connections 2024-01-27T23:25:54.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.520+0000 7fd047f79700 1 --2- 172.21.15.194:0/2871756934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0400f5830 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:54.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.520+0000 7fd047f79700 1 -- 172.21.15.194:0/2871756934 >> 172.21.15.194:0/2871756934 conn(0x7fd0400f0b70 msgr2=0x7fd0400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:54.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.520+0000 7fd047f79700 1 -- 172.21.15.194:0/2871756934 shutdown_connections 2024-01-27T23:25:54.510 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.520+0000 7fd047f79700 1 -- 172.21.15.194:0/2871756934 wait complete. 2024-01-27T23:25:54.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.521+0000 7fd047f79700 1 Processor -- start 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.521+0000 7fd047f79700 1 -- start start 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.521+0000 7fd047f79700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0401093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.521+0000 7fd047f79700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd040060280 con 0x7fd0400f5410 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.521+0000 7fd045d15700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0401093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.521+0000 7fd045d15700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0401093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:50938/0 (socket says 172.21.15.194:50938) 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.521+0000 7fd045d15700 1 -- 172.21.15.194:0/2392457435 learned_addr learned my addr 172.21.15.194:0/2392457435 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.522+0000 7fd045d15700 1 -- 172.21.15.194:0/2392457435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd034005040 con 0x7fd0400f5410 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.522+0000 7fd045d15700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0401093f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd034013510 tx=0x7fd03400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.522+0000 7fd0327fc700 1 -- 172.21.15.194:0/2392457435 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd034009320 con 0x7fd0400f5410 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.522+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd040060480 con 0x7fd0400f5410 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.522+0000 7fd0327fc700 1 -- 172.21.15.194:0/2392457435 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd034013f00 con 0x7fd0400f5410 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.522+0000 7fd0327fc700 1 -- 172.21.15.194:0/2392457435 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd034014d90 con 0x7fd0400f5410 2024-01-27T23:25:54.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.522+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd040109b20 con 0x7fd0400f5410 2024-01-27T23:25:54.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.523+0000 7fd0327fc700 1 -- 172.21.15.194:0/2392457435 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd0340263a0 con 0x7fd0400f5410 2024-01-27T23:25:54.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.523+0000 7fd0327fc700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd02c02ff60 0x7fd02c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:54.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.523+0000 7fd0327fc700 1 -- 172.21.15.194:0/2392457435 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd034041010 con 0x7fd0400f5410 2024-01-27T23:25:54.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.523+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd040059100 con 0x7fd0400f5410 2024-01-27T23:25:54.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.523+0000 7fd045514700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd02c02ff60 0x7fd02c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:54.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.524+0000 7fd045514700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd02c02ff60 0x7fd02c032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd03c0096a0 tx=0x7fd03c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:54.517 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.528+0000 7fd0327fc700 1 -- 172.21.15.194:0/2392457435 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd034012920 con 0x7fd0400f5410 2024-01-27T23:25:54.722 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.733+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd040045ad0 con 0x7fd0400f5410 2024-01-27T23:25:54.723 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.734+0000 7fd0327fc700 1 -- 172.21.15.194:0/2392457435 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd03402b020 con 0x7fd0400f5410 2024-01-27T23:25:54.723 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:25:54.723 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:25:54.724 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd02c02ff60 msgr2=0x7fd02c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd02c02ff60 0x7fd02c032420 secure :-1 s=READY pgs=14 cs=0 l=1 rev1=1 rx=0x7fd03c0096a0 tx=0x7fd03c006b40).stop 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 msgr2=0x7fd0401093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0401093f0 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7fd034013510 tx=0x7fd03400b040).stop 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 shutdown_connections 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd02c02ff60 0x7fd02c032420 unknown :-1 s=CLOSED pgs=14 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 --2- 172.21.15.194:0/2392457435 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd0400f5410 0x7fd0401093f0 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 >> 172.21.15.194:0/2392457435 conn(0x7fd0400f0b70 msgr2=0x7fd0400f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 shutdown_connections 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:54.735+0000 7fd047f79700 1 -- 172.21.15.194:0/2392457435 wait complete. 2024-01-27T23:25:54.725 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:25:55.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:55 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:54.734960+0000 mon.smithi121 (mon.0) 121 : audit [DBG] from='client.? 172.21.15.194:0/2392457435' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:25:56.123 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:25:56.124 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:25:56.336 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:55.262193+0000 mgr.smithi121.hcppvz (mgr.14162) 20 : cluster [DBG] pgmap v13: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:57.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.247+0000 7f13c3399700 1 -- 172.21.15.194:0/4154208965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 msgr2=0x7f13bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:57.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.247+0000 7f13c3399700 1 --2- 172.21.15.194:0/4154208965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc0f6930 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f13ac009fb0 tx=0x7f13ac009b30).stop 2024-01-27T23:25:57.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.247+0000 7f13c3399700 1 -- 172.21.15.194:0/4154208965 shutdown_connections 2024-01-27T23:25:57.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.247+0000 7f13c3399700 1 --2- 172.21.15.194:0/4154208965 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc0f6930 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:57.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.247+0000 7f13c3399700 1 -- 172.21.15.194:0/4154208965 >> 172.21.15.194:0/4154208965 conn(0x7f13bc0f1c70 msgr2=0x7f13bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:57.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.247+0000 7f13c3399700 1 -- 172.21.15.194:0/4154208965 shutdown_connections 2024-01-27T23:25:57.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.247+0000 7f13c3399700 1 -- 172.21.15.194:0/4154208965 wait complete. 2024-01-27T23:25:57.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.248+0000 7f13c3399700 1 Processor -- start 2024-01-27T23:25:57.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.248+0000 7f13c3399700 1 -- start start 2024-01-27T23:25:57.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.248+0000 7f13c3399700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:57.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.248+0000 7f13c3399700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f13bc106c20 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.248+0000 7f13c1135700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.248+0000 7f13c1135700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53812/0 (socket says 172.21.15.194:53812) 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.248+0000 7f13c1135700 1 -- 172.21.15.194:0/424137781 learned_addr learned my addr 172.21.15.194:0/424137781 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.249+0000 7f13c1135700 1 -- 172.21.15.194:0/424137781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f13ac005040 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.249+0000 7f13c1135700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc106730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f13ac014000 tx=0x7f13ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.249+0000 7f13b1ffb700 1 -- 172.21.15.194:0/424137781 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13ac0095f0 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.249+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f13bc106e20 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.249+0000 7f13b1ffb700 1 -- 172.21.15.194:0/424137781 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f13ac007e70 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.249+0000 7f13b1ffb700 1 -- 172.21.15.194:0/424137781 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f13ac014ea0 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.249+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f13bc1072b0 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.250+0000 7f13b1ffb700 1 -- 172.21.15.194:0/424137781 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f13ac025070 con 0x7f13bc0f6510 2024-01-27T23:25:57.240 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.250+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f13bc045ad0 con 0x7f13bc0f6510 2024-01-27T23:25:57.241 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.250+0000 7f13b1ffb700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f13a802ffb0 0x7f13a8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:25:57.241 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.250+0000 7f13b1ffb700 1 -- 172.21.15.194:0/424137781 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f13ac040a30 con 0x7f13bc0f6510 2024-01-27T23:25:57.241 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.250+0000 7f13c0934700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f13a802ffb0 0x7f13a8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:25:57.241 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.251+0000 7f13c0934700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f13a802ffb0 0x7f13a8032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f13b80096a0 tx=0x7f13b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:25:57.244 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.255+0000 7f13b1ffb700 1 -- 172.21.15.194:0/424137781 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f13ac0079c0 con 0x7f13bc0f6510 2024-01-27T23:25:57.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.459+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f13bc0fbf60 con 0x7f13bc0f6510 2024-01-27T23:25:57.449 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.460+0000 7f13b1ffb700 1 -- 172.21.15.194:0/424137781 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f13ac00f300 con 0x7f13bc0f6510 2024-01-27T23:25:57.449 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:25:57.449 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f13a802ffb0 msgr2=0x7f13a8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f13a802ffb0 0x7f13a8032470 secure :-1 s=READY pgs=15 cs=0 l=1 rev1=1 rx=0x7f13b80096a0 tx=0x7f13b8006b40).stop 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 msgr2=0x7f13bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc106730 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f13ac014000 tx=0x7f13ac00b040).stop 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 shutdown_connections 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f13a802ffb0 0x7f13a8032470 unknown :-1 s=CLOSED pgs=15 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 --2- 172.21.15.194:0/424137781 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f13bc0f6510 0x7f13bc106730 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.461+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 >> 172.21.15.194:0/424137781 conn(0x7f13bc0f1c70 msgr2=0x7f13bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:25:57.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.462+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 shutdown_connections 2024-01-27T23:25:57.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:25:57.462+0000 7f13c3399700 1 -- 172.21.15.194:0/424137781 wait complete. 2024-01-27T23:25:57.452 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:25:58.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:57.262434+0000 mgr.smithi121.hcppvz (mgr.14162) 21 : cluster [DBG] pgmap v14: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:25:58.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:25:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:25:57.461189+0000 mon.smithi121 (mon.0) 122 : audit [DBG] from='client.? 172.21.15.194:0/424137781' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:25:58.830 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:25:58.830 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:00.036 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.047+0000 7f4ef9c48700 1 -- 172.21.15.194:0/434538210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 msgr2=0x7f4ef40f5df0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:00.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.047+0000 7f4ef9c48700 1 --2- 172.21.15.194:0/434538210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef40f5df0 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f4ee4009fb0 tx=0x7f4ee4009b30).stop 2024-01-27T23:26:00.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.047+0000 7f4ef9c48700 1 -- 172.21.15.194:0/434538210 shutdown_connections 2024-01-27T23:26:00.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.047+0000 7f4ef9c48700 1 --2- 172.21.15.194:0/434538210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef40f5df0 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:00.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.047+0000 7f4ef9c48700 1 -- 172.21.15.194:0/434538210 >> 172.21.15.194:0/434538210 conn(0x7f4ef40f02e0 msgr2=0x7f4ef40f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:00.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.047+0000 7f4ef9c48700 1 -- 172.21.15.194:0/434538210 shutdown_connections 2024-01-27T23:26:00.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.047+0000 7f4ef9c48700 1 -- 172.21.15.194:0/434538210 wait complete. 2024-01-27T23:26:00.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.048+0000 7f4ef9c48700 1 Processor -- start 2024-01-27T23:26:00.038 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.048+0000 7f4ef9c48700 1 -- start start 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.048+0000 7f4ef9c48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef4106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.048+0000 7f4ef9c48700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ef4106c10 con 0x7f4ef40f59d0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4ef37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef4106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4ef37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef4106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53838/0 (socket says 172.21.15.194:53838) 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4ef37fe700 1 -- 172.21.15.194:0/896569469 learned_addr learned my addr 172.21.15.194:0/896569469 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4ef37fe700 1 -- 172.21.15.194:0/896569469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee4005040 con 0x7f4ef40f59d0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4ef37fe700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef4106720 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4ee4013510 tx=0x7f4ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4edbfff700 1 -- 172.21.15.194:0/896569469 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee4009320 con 0x7f4ef40f59d0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ef4106e10 con 0x7f4ef40f59d0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4edbfff700 1 -- 172.21.15.194:0/896569469 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ee4013f00 con 0x7f4ef40f59d0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4edbfff700 1 -- 172.21.15.194:0/896569469 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee4014d90 con 0x7f4ef40f59d0 2024-01-27T23:26:00.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.049+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ef41072a0 con 0x7f4ef40f59d0 2024-01-27T23:26:00.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.050+0000 7f4edbfff700 1 -- 172.21.15.194:0/896569469 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ee40263a0 con 0x7f4ef40f59d0 2024-01-27T23:26:00.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.050+0000 7f4edbfff700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4edc02ff60 0x7f4edc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:00.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.050+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ef4045ad0 con 0x7f4ef40f59d0 2024-01-27T23:26:00.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.050+0000 7f4edbfff700 1 -- 172.21.15.194:0/896569469 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ee4041400 con 0x7f4ef40f59d0 2024-01-27T23:26:00.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.050+0000 7f4ef2ffd700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4edc02ff60 0x7f4edc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:00.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.051+0000 7f4ef2ffd700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4edc02ff60 0x7f4edc032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4ee80096a0 tx=0x7f4ee8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:00.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.055+0000 7f4edbfff700 1 -- 172.21.15.194:0/896569469 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ee4012510 con 0x7f4ef40f59d0 2024-01-27T23:26:00.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.258+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ef40fc040 con 0x7f4ef40f59d0 2024-01-27T23:26:00.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.259+0000 7f4edbfff700 1 -- 172.21.15.194:0/896569469 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4ee4012330 con 0x7f4ef40f59d0 2024-01-27T23:26:00.248 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:00.248 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:00.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4edc02ff60 msgr2=0x7f4edc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:00.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4edc02ff60 0x7f4edc032420 secure :-1 s=READY pgs=16 cs=0 l=1 rev1=1 rx=0x7f4ee80096a0 tx=0x7f4ee8006b40).stop 2024-01-27T23:26:00.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 msgr2=0x7f4ef4106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:00.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef4106720 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7f4ee4013510 tx=0x7f4ee400b040).stop 2024-01-27T23:26:00.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 shutdown_connections 2024-01-27T23:26:00.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4edc02ff60 0x7f4edc032420 unknown :-1 s=CLOSED pgs=16 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:00.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 --2- 172.21.15.194:0/896569469 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ef40f59d0 0x7f4ef4106720 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:00.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.260+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 >> 172.21.15.194:0/896569469 conn(0x7f4ef40f02e0 msgr2=0x7f4ef40f0dd0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:00.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.261+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 shutdown_connections 2024-01-27T23:26:00.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:00.261+0000 7f4ef9c48700 1 -- 172.21.15.194:0/896569469 wait complete. 2024-01-27T23:26:00.251 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:00.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:25:59.262755+0000 mgr.smithi121.hcppvz (mgr.14162) 22 : cluster [DBG] pgmap v15: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:00.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:00.260240+0000 mon.smithi121 (mon.0) 123 : audit [DBG] from='client.? 172.21.15.194:0/896569469' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:01.638 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:01.638 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:02.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:01.263037+0000 mgr.smithi121.hcppvz (mgr.14162) 23 : cluster [DBG] pgmap v16: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:02.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.842+0000 7f57e912e700 1 -- 172.21.15.194:0/804238562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 msgr2=0x7f57e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:02.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.842+0000 7f57e912e700 1 --2- 172.21.15.194:0/804238562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e40f5830 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f57cc009fb0 tx=0x7f57cc009b30).stop 2024-01-27T23:26:02.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.842+0000 7f57e912e700 1 -- 172.21.15.194:0/804238562 shutdown_connections 2024-01-27T23:26:02.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.842+0000 7f57e912e700 1 --2- 172.21.15.194:0/804238562 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e40f5830 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.842+0000 7f57e912e700 1 -- 172.21.15.194:0/804238562 >> 172.21.15.194:0/804238562 conn(0x7f57e40f0b70 msgr2=0x7f57e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.842+0000 7f57e912e700 1 -- 172.21.15.194:0/804238562 shutdown_connections 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.842+0000 7f57e912e700 1 -- 172.21.15.194:0/804238562 wait complete. 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.843+0000 7f57e912e700 1 Processor -- start 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.843+0000 7f57e912e700 1 -- start start 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.843+0000 7f57e912e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e41093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.843+0000 7f57e912e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57e4060280 con 0x7f57e40f5410 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.843+0000 7f57e2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e41093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.843+0000 7f57e2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e41093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53844/0 (socket says 172.21.15.194:53844) 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.843+0000 7f57e2d9d700 1 -- 172.21.15.194:0/2685362945 learned_addr learned my addr 172.21.15.194:0/2685362945 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.844+0000 7f57e2d9d700 1 -- 172.21.15.194:0/2685362945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc005040 con 0x7f57e40f5410 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.844+0000 7f57e2d9d700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e41093f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f57cc013510 tx=0x7f57cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.844+0000 7f57db7fe700 1 -- 172.21.15.194:0/2685362945 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57cc009320 con 0x7f57e40f5410 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.844+0000 7f57db7fe700 1 -- 172.21.15.194:0/2685362945 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57cc013f00 con 0x7f57e40f5410 2024-01-27T23:26:02.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.844+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57e4060480 con 0x7f57e40f5410 2024-01-27T23:26:02.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.844+0000 7f57db7fe700 1 -- 172.21.15.194:0/2685362945 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57cc014d90 con 0x7f57e40f5410 2024-01-27T23:26:02.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.844+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57e4109b20 con 0x7f57e40f5410 2024-01-27T23:26:02.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.845+0000 7f57db7fe700 1 -- 172.21.15.194:0/2685362945 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57cc0095f0 con 0x7f57e40f5410 2024-01-27T23:26:02.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.845+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57e4045ad0 con 0x7f57e40f5410 2024-01-27T23:26:02.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.845+0000 7f57db7fe700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57d002ff60 0x7f57d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:02.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.845+0000 7f57db7fe700 1 -- 172.21.15.194:0/2685362945 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57cc041000 con 0x7f57e40f5410 2024-01-27T23:26:02.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.845+0000 7f57e259c700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57d002ff60 0x7f57d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:02.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.846+0000 7f57e259c700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57d002ff60 0x7f57d0032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f57d40096a0 tx=0x7f57d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:02.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:02.850+0000 7f57db7fe700 1 -- 172.21.15.194:0/2685362945 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57cc014420 con 0x7f57e40f5410 2024-01-27T23:26:03.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.056+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57e410a340 con 0x7f57e40f5410 2024-01-27T23:26:03.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.057+0000 7f57db7fe700 1 -- 172.21.15.194:0/2685362945 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f57cc01d7f0 con 0x7f57e40f5410 2024-01-27T23:26:03.046 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:03.046 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:03.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57d002ff60 msgr2=0x7f57d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:03.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57d002ff60 0x7f57d0032420 secure :-1 s=READY pgs=17 cs=0 l=1 rev1=1 rx=0x7f57d40096a0 tx=0x7f57d4006b40).stop 2024-01-27T23:26:03.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 msgr2=0x7f57e41093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:03.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e41093f0 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f57cc013510 tx=0x7f57cc00b040).stop 2024-01-27T23:26:03.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 shutdown_connections 2024-01-27T23:26:03.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57d002ff60 0x7f57d0032420 unknown :-1 s=CLOSED pgs=17 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:03.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 --2- 172.21.15.194:0/2685362945 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57e40f5410 0x7f57e41093f0 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:03.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.058+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 >> 172.21.15.194:0/2685362945 conn(0x7f57e40f0b70 msgr2=0x7f57e40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:03.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.059+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 shutdown_connections 2024-01-27T23:26:03.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:03.059+0000 7f57e912e700 1 -- 172.21.15.194:0/2685362945 wait complete. 2024-01-27T23:26:03.049 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:03.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:03 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:03.058295+0000 mon.smithi121 (mon.0) 124 : audit [DBG] from='client.? 172.21.15.194:0/2685362945' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:04.451 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:04.451 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:04.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:03.263281+0000 mgr.smithi121.hcppvz (mgr.14162) 24 : cluster [DBG] pgmap v17: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:05.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.620+0000 7f792fb8e700 1 -- 172.21.15.194:0/247483733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 msgr2=0x7f79280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:05.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.620+0000 7f792fb8e700 1 --2- 172.21.15.194:0/247483733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f79280f6930 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7f791c009fb0 tx=0x7f791c009b30).stop 2024-01-27T23:26:05.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.620+0000 7f792fb8e700 1 -- 172.21.15.194:0/247483733 shutdown_connections 2024-01-27T23:26:05.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.620+0000 7f792fb8e700 1 --2- 172.21.15.194:0/247483733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f79280f6930 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:05.611 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.620+0000 7f792fb8e700 1 -- 172.21.15.194:0/247483733 >> 172.21.15.194:0/247483733 conn(0x7f79280f1c70 msgr2=0x7f79280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792fb8e700 1 -- 172.21.15.194:0/247483733 shutdown_connections 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792fb8e700 1 -- 172.21.15.194:0/247483733 wait complete. 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792fb8e700 1 Processor -- start 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792fb8e700 1 -- start start 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792fb8e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792fb8e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7928069e20 con 0x7f79280f6510 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792d92a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792d92a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53036/0 (socket says 172.21.15.194:53036) 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.621+0000 7f792d92a700 1 -- 172.21.15.194:0/548556148 learned_addr learned my addr 172.21.15.194:0/548556148 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.622+0000 7f792d92a700 1 -- 172.21.15.194:0/548556148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f791c005040 con 0x7f79280f6510 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.622+0000 7f792d92a700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f791c014000 tx=0x7f791c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.622+0000 7f791a7fc700 1 -- 172.21.15.194:0/548556148 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791c0095f0 con 0x7f79280f6510 2024-01-27T23:26:05.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.622+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7928069fc0 con 0x7f79280f6510 2024-01-27T23:26:05.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.622+0000 7f791a7fc700 1 -- 172.21.15.194:0/548556148 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f791c007e70 con 0x7f79280f6510 2024-01-27T23:26:05.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.622+0000 7f791a7fc700 1 -- 172.21.15.194:0/548556148 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f791c014ea0 con 0x7f79280f6510 2024-01-27T23:26:05.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.622+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f792806a450 con 0x7f79280f6510 2024-01-27T23:26:05.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.623+0000 7f791a7fc700 1 -- 172.21.15.194:0/548556148 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f791c025070 con 0x7f79280f6510 2024-01-27T23:26:05.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.623+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7928059100 con 0x7f79280f6510 2024-01-27T23:26:05.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.623+0000 7f791a7fc700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f791402ff60 0x7f7914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:05.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.623+0000 7f791a7fc700 1 -- 172.21.15.194:0/548556148 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f791c040920 con 0x7f79280f6510 2024-01-27T23:26:05.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.623+0000 7f792d129700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f791402ff60 0x7f7914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:05.614 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.623+0000 7f792d129700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f791402ff60 0x7f7914032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f79240096a0 tx=0x7f7924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:05.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.628+0000 7f791a7fc700 1 -- 172.21.15.194:0/548556148 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f791c01d4e0 con 0x7f79280f6510 2024-01-27T23:26:05.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.834+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7928045ad0 con 0x7f79280f6510 2024-01-27T23:26:05.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.835+0000 7f791a7fc700 1 -- 172.21.15.194:0/548556148 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f791c00f300 con 0x7f79280f6510 2024-01-27T23:26:05.824 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:05.824 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f791402ff60 msgr2=0x7f7914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f791402ff60 0x7f7914032420 secure :-1 s=READY pgs=18 cs=0 l=1 rev1=1 rx=0x7f79240096a0 tx=0x7f7924006b40).stop 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 msgr2=0x7f792806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f791c014000 tx=0x7f791c00b040).stop 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 shutdown_connections 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f791402ff60 0x7f7914032420 unknown :-1 s=CLOSED pgs=18 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 --2- 172.21.15.194:0/548556148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f79280f6510 0x7f792806b9a0 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 >> 172.21.15.194:0/548556148 conn(0x7f79280f1c70 msgr2=0x7f79280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.836+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 shutdown_connections 2024-01-27T23:26:05.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:05.837+0000 7f792fb8e700 1 -- 172.21.15.194:0/548556148 wait complete. 2024-01-27T23:26:05.827 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:06.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:05.263481+0000 mgr.smithi121.hcppvz (mgr.14162) 25 : cluster [DBG] pgmap v18: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:06.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:05.836207+0000 mon.smithi121 (mon.0) 125 : audit [DBG] from='client.? 172.21.15.194:0/548556148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:07.211 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:07.212 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.379+0000 7ff4cece0700 1 -- 172.21.15.194:0/644985652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 msgr2=0x7ff4c8069ec0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.379+0000 7ff4cece0700 1 --2- 172.21.15.194:0/644985652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c8069ec0 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7ff4bc009fb0 tx=0x7ff4bc009b30).stop 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.379+0000 7ff4cece0700 1 -- 172.21.15.194:0/644985652 shutdown_connections 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.379+0000 7ff4cece0700 1 --2- 172.21.15.194:0/644985652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c8069ec0 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.379+0000 7ff4cece0700 1 -- 172.21.15.194:0/644985652 >> 172.21.15.194:0/644985652 conn(0x7ff4c80f1c70 msgr2=0x7ff4c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.379+0000 7ff4cece0700 1 -- 172.21.15.194:0/644985652 shutdown_connections 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.379+0000 7ff4cece0700 1 -- 172.21.15.194:0/644985652 wait complete. 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.380+0000 7ff4cece0700 1 Processor -- start 2024-01-27T23:26:08.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.380+0000 7ff4cece0700 1 -- start start 2024-01-27T23:26:08.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.380+0000 7ff4cece0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c81088f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:08.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.380+0000 7ff4cece0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff4c8108de0 con 0x7ff4c806b9c0 2024-01-27T23:26:08.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.380+0000 7ff4cca7c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c81088f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:08.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.380+0000 7ff4cca7c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c81088f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53054/0 (socket says 172.21.15.194:53054) 2024-01-27T23:26:08.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.380+0000 7ff4cca7c700 1 -- 172.21.15.194:0/3366748397 learned_addr learned my addr 172.21.15.194:0/3366748397 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.381+0000 7ff4cca7c700 1 -- 172.21.15.194:0/3366748397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff4bc005040 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.381+0000 7ff4cca7c700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c81088f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff4bc013510 tx=0x7ff4bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.381+0000 7ff4c57fa700 1 -- 172.21.15.194:0/3366748397 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4bc009320 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.381+0000 7ff4c57fa700 1 -- 172.21.15.194:0/3366748397 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff4bc013f00 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.381+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff4c8108fe0 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.381+0000 7ff4c57fa700 1 -- 172.21.15.194:0/3366748397 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff4bc014dc0 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.381+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff4c8109470 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.382+0000 7ff4c57fa700 1 -- 172.21.15.194:0/3366748397 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff4bc01d3d0 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.382+0000 7ff4c57fa700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff4b802ff60 0x7ff4b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.382+0000 7ff4c57fa700 1 -- 172.21.15.194:0/3366748397 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff4bc041060 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.382+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff4c8045ad0 con 0x7ff4c806b9c0 2024-01-27T23:26:08.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.382+0000 7ff4c7fff700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff4b802ff60 0x7ff4b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:08.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.383+0000 7ff4c7fff700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff4b802ff60 0x7ff4b8032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff4b40096a0 tx=0x7ff4b4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:08.376 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.387+0000 7ff4c57fa700 1 -- 172.21.15.194:0/3366748397 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff4bc01db80 con 0x7ff4c806b9c0 2024-01-27T23:26:08.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.594+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff4c810a320 con 0x7ff4c806b9c0 2024-01-27T23:26:08.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.595+0000 7ff4c57fa700 1 -- 172.21.15.194:0/3366748397 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff4bc02b020 con 0x7ff4c806b9c0 2024-01-27T23:26:08.584 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:08.584 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff4b802ff60 msgr2=0x7ff4b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff4b802ff60 0x7ff4b8032420 secure :-1 s=READY pgs=19 cs=0 l=1 rev1=1 rx=0x7ff4b40096a0 tx=0x7ff4b4006b40).stop 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 msgr2=0x7ff4c81088f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c81088f0 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7ff4bc013510 tx=0x7ff4bc00b040).stop 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 shutdown_connections 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff4b802ff60 0x7ff4b8032420 unknown :-1 s=CLOSED pgs=19 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 --2- 172.21.15.194:0/3366748397 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff4c806b9c0 0x7ff4c81088f0 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:08.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 >> 172.21.15.194:0/3366748397 conn(0x7ff4c80f1c70 msgr2=0x7ff4c80f2740 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:08.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 shutdown_connections 2024-01-27T23:26:08.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:08.596+0000 7ff4cece0700 1 -- 172.21.15.194:0/3366748397 wait complete. 2024-01-27T23:26:08.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:07.263685+0000 mgr.smithi121.hcppvz (mgr.14162) 26 : cluster [DBG] pgmap v19: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:08.587 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:09.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:08.596080+0000 mon.smithi121 (mon.0) 126 : audit [DBG] from='client.? 172.21.15.194:0/3366748397' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:10.205 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:10.206 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:10.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:09.264044+0000 mgr.smithi121.hcppvz (mgr.14162) 27 : cluster [DBG] pgmap v20: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:12.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.287+0000 7f1790668700 1 -- 172.21.15.194:0/1590140454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 msgr2=0x7f17880f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:12.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.287+0000 7f1790668700 1 --2- 172.21.15.194:0/1590140454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f17880f5830 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f1780009960 tx=0x7f1780004d10).stop 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.287+0000 7f1790668700 1 -- 172.21.15.194:0/1590140454 shutdown_connections 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.287+0000 7f1790668700 1 --2- 172.21.15.194:0/1590140454 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f17880f5830 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.287+0000 7f1790668700 1 -- 172.21.15.194:0/1590140454 >> 172.21.15.194:0/1590140454 conn(0x7f17880f0b70 msgr2=0x7f17880f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.287+0000 7f1790668700 1 -- 172.21.15.194:0/1590140454 shutdown_connections 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.287+0000 7f1790668700 1 -- 172.21.15.194:0/1590140454 wait complete. 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.288+0000 7f1790668700 1 Processor -- start 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.288+0000 7f1790668700 1 -- start start 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.288+0000 7f1790668700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f1788105690 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.288+0000 7f1790668700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1788105b80 con 0x7f17880f5410 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.288+0000 7f178e404700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f1788105690 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.288+0000 7f178e404700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f1788105690 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53074/0 (socket says 172.21.15.194:53074) 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.288+0000 7f178e404700 1 -- 172.21.15.194:0/1698206798 learned_addr learned my addr 172.21.15.194:0/1698206798 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:12.279 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.289+0000 7f178e404700 1 -- 172.21.15.194:0/1698206798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1780009640 con 0x7f17880f5410 2024-01-27T23:26:12.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.289+0000 7f178e404700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f1788105690 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1780004a10 tx=0x7f1780004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:12.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.289+0000 7f1786ffd700 1 -- 172.21.15.194:0/1698206798 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1780018070 con 0x7f17880f5410 2024-01-27T23:26:12.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.289+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1788105d80 con 0x7f17880f5410 2024-01-27T23:26:12.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.289+0000 7f1786ffd700 1 -- 172.21.15.194:0/1698206798 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1780007e70 con 0x7f17880f5410 2024-01-27T23:26:12.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.289+0000 7f1786ffd700 1 -- 172.21.15.194:0/1698206798 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f178001d420 con 0x7f17880f5410 2024-01-27T23:26:12.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.289+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1788106210 con 0x7f17880f5410 2024-01-27T23:26:12.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.290+0000 7f1786ffd700 1 -- 172.21.15.194:0/1698206798 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f178001d580 con 0x7f17880f5410 2024-01-27T23:26:12.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.290+0000 7f1786ffd700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f177002ff60 0x7f1770032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:12.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.290+0000 7f1786ffd700 1 -- 172.21.15.194:0/1698206798 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1780041610 con 0x7f17880f5410 2024-01-27T23:26:12.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.290+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1788045ad0 con 0x7f17880f5410 2024-01-27T23:26:12.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.290+0000 7f178dc03700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f177002ff60 0x7f1770032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:12.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.291+0000 7f178dc03700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f177002ff60 0x7f1770032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f177c0096a0 tx=0x7f177c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:12.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.295+0000 7f1786ffd700 1 -- 172.21.15.194:0/1698206798 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1780027440 con 0x7f17880f5410 2024-01-27T23:26:12.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.502+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1788106f90 con 0x7f17880f5410 2024-01-27T23:26:12.491 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.502+0000 7f1786ffd700 1 -- 172.21.15.194:0/1698206798 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f178001bb60 con 0x7f17880f5410 2024-01-27T23:26:12.491 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:12.492 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:12.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f177002ff60 msgr2=0x7f1770032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:12.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f177002ff60 0x7f1770032420 secure :-1 s=READY pgs=20 cs=0 l=1 rev1=1 rx=0x7f177c0096a0 tx=0x7f177c006b40).stop 2024-01-27T23:26:12.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 msgr2=0x7f1788105690 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:12.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f1788105690 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7f1780004a10 tx=0x7f1780004d10).stop 2024-01-27T23:26:12.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 shutdown_connections 2024-01-27T23:26:12.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f177002ff60 0x7f1770032420 unknown :-1 s=CLOSED pgs=20 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:12.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 --2- 172.21.15.194:0/1698206798 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17880f5410 0x7f1788105690 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:12.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 >> 172.21.15.194:0/1698206798 conn(0x7f17880f0b70 msgr2=0x7f17880f1720 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:12.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 shutdown_connections 2024-01-27T23:26:12.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:12.504+0000 7f1790668700 1 -- 172.21.15.194:0/1698206798 wait complete. 2024-01-27T23:26:12.494 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:12.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:11.264333+0000 mgr.smithi121.hcppvz (mgr.14162) 28 : cluster [DBG] pgmap v21: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:13.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:13 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:12.503532+0000 mon.smithi121 (mon.0) 127 : audit [DBG] from='client.? 172.21.15.194:0/1698206798' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:13.867 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:13.867 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:14.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:13.264572+0000 mgr.smithi121.hcppvz (mgr.14162) 29 : cluster [DBG] pgmap v22: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.054+0000 7fa205f3d700 1 -- 172.21.15.194:0/3213047007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 msgr2=0x7fa2000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.054+0000 7fa205f3d700 1 --2- 172.21.15.194:0/3213047007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa2000f5830 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7fa1f0009fb0 tx=0x7fa1f0009b30).stop 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.054+0000 7fa205f3d700 1 -- 172.21.15.194:0/3213047007 shutdown_connections 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.054+0000 7fa205f3d700 1 --2- 172.21.15.194:0/3213047007 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa2000f5830 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.054+0000 7fa205f3d700 1 -- 172.21.15.194:0/3213047007 >> 172.21.15.194:0/3213047007 conn(0x7fa2000f0b70 msgr2=0x7fa2000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.054+0000 7fa205f3d700 1 -- 172.21.15.194:0/3213047007 shutdown_connections 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.054+0000 7fa205f3d700 1 -- 172.21.15.194:0/3213047007 wait complete. 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.055+0000 7fa205f3d700 1 Processor -- start 2024-01-27T23:26:15.045 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.055+0000 7fa205f3d700 1 -- start start 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.055+0000 7fa205f3d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa200109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.055+0000 7fa205f3d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa200060220 con 0x7fa2000f5410 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.055+0000 7fa1ff7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa200109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.055+0000 7fa1ff7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa200109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53106/0 (socket says 172.21.15.194:53106) 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.055+0000 7fa1ff7fe700 1 -- 172.21.15.194:0/4095637014 learned_addr learned my addr 172.21.15.194:0/4095637014 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.056+0000 7fa1ff7fe700 1 -- 172.21.15.194:0/4095637014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa1f0005040 con 0x7fa2000f5410 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.056+0000 7fa1ff7fe700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa200109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa1f0013510 tx=0x7fa1f000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.056+0000 7fa1e7fff700 1 -- 172.21.15.194:0/4095637014 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1f0009320 con 0x7fa2000f5410 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.056+0000 7fa1e7fff700 1 -- 172.21.15.194:0/4095637014 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa1f0013f00 con 0x7fa2000f5410 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.056+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa200060420 con 0x7fa2000f5410 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.056+0000 7fa1e7fff700 1 -- 172.21.15.194:0/4095637014 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa1f0014d90 con 0x7fa2000f5410 2024-01-27T23:26:15.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.056+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa200109b00 con 0x7fa2000f5410 2024-01-27T23:26:15.047 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.057+0000 7fa1e7fff700 1 -- 172.21.15.194:0/4095637014 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1f00263a0 con 0x7fa2000f5410 2024-01-27T23:26:15.047 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.057+0000 7fa1e7fff700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa1e802ff60 0x7fa1e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:15.047 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.057+0000 7fa1e7fff700 1 -- 172.21.15.194:0/4095637014 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1f0041000 con 0x7fa2000f5410 2024-01-27T23:26:15.047 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.057+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa200059100 con 0x7fa2000f5410 2024-01-27T23:26:15.047 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.057+0000 7fa1feffd700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa1e802ff60 0x7fa1e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:15.047 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.057+0000 7fa1feffd700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa1e802ff60 0x7fa1e8032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa1f40096a0 tx=0x7fa1f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:15.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.062+0000 7fa1e7fff700 1 -- 172.21.15.194:0/4095637014 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa1f0012920 con 0x7fa2000f5410 2024-01-27T23:26:15.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.269+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa200045ad0 con 0x7fa2000f5410 2024-01-27T23:26:15.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.269+0000 7fa1e7fff700 1 -- 172.21.15.194:0/4095637014 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa1f002b020 con 0x7fa2000f5410 2024-01-27T23:26:15.259 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:15.259 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:15.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa1e802ff60 msgr2=0x7fa1e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:15.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa1e802ff60 0x7fa1e8032420 secure :-1 s=READY pgs=21 cs=0 l=1 rev1=1 rx=0x7fa1f40096a0 tx=0x7fa1f4006b40).stop 2024-01-27T23:26:15.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 msgr2=0x7fa200109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:15.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa200109450 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7fa1f0013510 tx=0x7fa1f000b040).stop 2024-01-27T23:26:15.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 shutdown_connections 2024-01-27T23:26:15.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa1e802ff60 0x7fa1e8032420 unknown :-1 s=CLOSED pgs=21 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:15.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 --2- 172.21.15.194:0/4095637014 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2000f5410 0x7fa200109450 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:15.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 >> 172.21.15.194:0/4095637014 conn(0x7fa2000f0b70 msgr2=0x7fa2000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:15.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 shutdown_connections 2024-01-27T23:26:15.261 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:15.271+0000 7fa205f3d700 1 -- 172.21.15.194:0/4095637014 wait complete. 2024-01-27T23:26:15.261 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:16.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:15.264906+0000 mgr.smithi121.hcppvz (mgr.14162) 30 : cluster [DBG] pgmap v23: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:16.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:15.270825+0000 mon.smithi121 (mon.0) 128 : audit [DBG] from='client.? 172.21.15.194:0/4095637014' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:16.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:16.647 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:18.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:17.265192+0000 mgr.smithi121.hcppvz (mgr.14162) 31 : cluster [DBG] pgmap v24: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:18.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:17.391981+0000 mon.smithi121 (mon.0) 129 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:26:18.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:17.400382+0000 mon.smithi121 (mon.0) 130 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:26:20.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:19.265496+0000 mgr.smithi121.hcppvz (mgr.14162) 32 : cluster [DBG] pgmap v25: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 -- 172.21.15.194:0/813080482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 msgr2=0x7f933c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 --2- 172.21.15.194:0/813080482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c0f66c0 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f9330009fb0 tx=0x7f9330009b30).stop 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 -- 172.21.15.194:0/813080482 shutdown_connections 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 --2- 172.21.15.194:0/813080482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c0f66c0 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 -- 172.21.15.194:0/813080482 >> 172.21.15.194:0/813080482 conn(0x7f933c0f1aa0 msgr2=0x7f933c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 -- 172.21.15.194:0/813080482 shutdown_connections 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 -- 172.21.15.194:0/813080482 wait complete. 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.027+0000 7f933b59e700 1 Processor -- start 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933b59e700 1 -- start start 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933b59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:21.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933b59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f933c1069a0 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48734/0 (socket says 172.21.15.194:48734) 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933a59c700 1 -- 172.21.15.194:0/3715476701 learned_addr learned my addr 172.21.15.194:0/3715476701 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933a59c700 1 -- 172.21.15.194:0/3715476701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9330005040 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f933a59c700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c1064b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9330013510 tx=0x7f933000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.028+0000 7f932affd700 1 -- 172.21.15.194:0/3715476701 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9330009320 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.029+0000 7f932affd700 1 -- 172.21.15.194:0/3715476701 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9330013f00 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.029+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f933c106ba0 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.029+0000 7f932affd700 1 -- 172.21.15.194:0/3715476701 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9330014dc0 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.029+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f933c107030 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.029+0000 7f932affd700 1 -- 172.21.15.194:0/3715476701 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9330014420 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.029+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f933c045ad0 con 0x7f933c0f62a0 2024-01-27T23:26:21.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.030+0000 7f932affd700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f932402ffb0 0x7f9324032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.030+0000 7f932affd700 1 -- 172.21.15.194:0/3715476701 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9330040ff0 con 0x7f933c0f62a0 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.030+0000 7f9339d9b700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f932402ffb0 0x7f9324032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.030+0000 7f9339d9b700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f932402ffb0 0x7f9324032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f932c0096a0 tx=0x7f932c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.034+0000 7f932affd700 1 -- 172.21.15.194:0/3715476701 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f933006d2b0 con 0x7f933c0f62a0 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.239+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f933c107ee0 con 0x7f933c0f62a0 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.240+0000 7f932affd700 1 -- 172.21.15.194:0/3715476701 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f93300095f0 con 0x7f933c0f62a0 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f932402ffb0 msgr2=0x7f9324032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f932402ffb0 0x7f9324032470 secure :-1 s=READY pgs=22 cs=0 l=1 rev1=1 rx=0x7f932c0096a0 tx=0x7f932c006b40).stop 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 msgr2=0x7f933c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c1064b0 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f9330013510 tx=0x7f933000b040).stop 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 shutdown_connections 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f932402ffb0 0x7f9324032470 unknown :-1 s=CLOSED pgs=22 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 --2- 172.21.15.194:0/3715476701 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f933c0f62a0 0x7f933c1064b0 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:21.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 >> 172.21.15.194:0/3715476701 conn(0x7f933c0f1aa0 msgr2=0x7f933c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:21.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 shutdown_connections 2024-01-27T23:26:21.260 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:21.242+0000 7f933b59e700 1 -- 172.21.15.194:0/3715476701 wait complete. 2024-01-27T23:26:21.260 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:21.260 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:21.260 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:21.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:21 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:21.241450+0000 mon.smithi121 (mon.0) 131 : audit [DBG] from='client.? 172.21.15.194:0/3715476701' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:22.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:21.265742+0000 mgr.smithi121.hcppvz (mgr.14162) 33 : cluster [DBG] pgmap v26: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:22.648 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:22.649 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:23.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.792+0000 7fdf691cf700 1 -- 172.21.15.194:0/3292781895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 msgr2=0x7fdf640f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:23.781 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.792+0000 7fdf691cf700 1 --2- 172.21.15.194:0/3292781895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf640f9190 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fdf54009fb0 tx=0x7fdf54009b30).stop 2024-01-27T23:26:23.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.792+0000 7fdf691cf700 1 -- 172.21.15.194:0/3292781895 shutdown_connections 2024-01-27T23:26:23.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.792+0000 7fdf691cf700 1 --2- 172.21.15.194:0/3292781895 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf640f9190 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:23.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.792+0000 7fdf691cf700 1 -- 172.21.15.194:0/3292781895 >> 172.21.15.194:0/3292781895 conn(0x7fdf640f0b70 msgr2=0x7fdf640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:23.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.792+0000 7fdf691cf700 1 -- 172.21.15.194:0/3292781895 shutdown_connections 2024-01-27T23:26:23.782 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.792+0000 7fdf691cf700 1 -- 172.21.15.194:0/3292781895 wait complete. 2024-01-27T23:26:23.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf691cf700 1 Processor -- start 2024-01-27T23:26:23.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf691cf700 1 -- start start 2024-01-27T23:26:23.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf691cf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf641088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:23.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf691cf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fdf64108da0 con 0x7fdf640601e0 2024-01-27T23:26:23.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf62d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf641088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:23.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf62d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf641088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48750/0 (socket says 172.21.15.194:48750) 2024-01-27T23:26:23.784 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf62d9d700 1 -- 172.21.15.194:0/3555697 learned_addr learned my addr 172.21.15.194:0/3555697 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf62d9d700 1 -- 172.21.15.194:0/3555697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fdf54005040 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.793+0000 7fdf62d9d700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf641088b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fdf54004dd0 tx=0x7fdf54009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.794+0000 7fdf4b7fe700 1 -- 172.21.15.194:0/3555697 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf54018070 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.794+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fdf64108fa0 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.794+0000 7fdf4b7fe700 1 -- 172.21.15.194:0/3555697 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fdf54013410 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.794+0000 7fdf4b7fe700 1 -- 172.21.15.194:0/3555697 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fdf5401c3c0 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.794+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fdf64109430 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.794+0000 7fdf4b7fe700 1 -- 172.21.15.194:0/3555697 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fdf5401c630 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.795+0000 7fdf4b7fe700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdf4c02ff60 0x7fdf4c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.795+0000 7fdf4b7fe700 1 -- 172.21.15.194:0/3555697 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fdf54041390 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.795+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fdf64045ad0 con 0x7fdf640601e0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.795+0000 7fdf6259c700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdf4c02ff60 0x7fdf4c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:23.785 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.795+0000 7fdf6259c700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdf4c02ff60 0x7fdf4c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fdf580096a0 tx=0x7fdf58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:23.788 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:23.799+0000 7fdf4b7fe700 1 -- 172.21.15.194:0/3555697 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fdf5401ca30 con 0x7fdf640601e0 2024-01-27T23:26:24.000 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.011+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fdf6410a170 con 0x7fdf640601e0 2024-01-27T23:26:24.000 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.011+0000 7fdf4b7fe700 1 -- 172.21.15.194:0/3555697 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fdf5401b9c0 con 0x7fdf640601e0 2024-01-27T23:26:24.001 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:24.001 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:24.002 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdf4c02ff60 msgr2=0x7fdf4c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdf4c02ff60 0x7fdf4c032420 secure :-1 s=READY pgs=23 cs=0 l=1 rev1=1 rx=0x7fdf580096a0 tx=0x7fdf58006b40).stop 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 msgr2=0x7fdf641088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf641088b0 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7fdf54004dd0 tx=0x7fdf54009b30).stop 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 shutdown_connections 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fdf4c02ff60 0x7fdf4c032420 unknown :-1 s=CLOSED pgs=23 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 --2- 172.21.15.194:0/3555697 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fdf640601e0 0x7fdf641088b0 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 >> 172.21.15.194:0/3555697 conn(0x7fdf640f0b70 msgr2=0x7fdf640f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 shutdown_connections 2024-01-27T23:26:24.003 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:24.013+0000 7fdf691cf700 1 -- 172.21.15.194:0/3555697 wait complete. 2024-01-27T23:26:24.004 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:24.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:23.266081+0000 mgr.smithi121.hcppvz (mgr.14162) 34 : cluster [DBG] pgmap v27: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:24.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:24.012767+0000 mon.smithi121 (mon.0) 132 : audit [DBG] from='client.? 172.21.15.194:0/3555697' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:25.393 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:25.393 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.591+0000 7f0281e05700 1 -- 172.21.15.194:0/1261154879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 msgr2=0x7f027c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.591+0000 7f0281e05700 1 --2- 172.21.15.194:0/1261154879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c0f5830 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7f0264009fb0 tx=0x7f0264009b30).stop 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.591+0000 7f0281e05700 1 -- 172.21.15.194:0/1261154879 shutdown_connections 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.591+0000 7f0281e05700 1 --2- 172.21.15.194:0/1261154879 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c0f5830 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.591+0000 7f0281e05700 1 -- 172.21.15.194:0/1261154879 >> 172.21.15.194:0/1261154879 conn(0x7f027c0f0b70 msgr2=0x7f027c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.591+0000 7f0281e05700 1 -- 172.21.15.194:0/1261154879 shutdown_connections 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.591+0000 7f0281e05700 1 -- 172.21.15.194:0/1261154879 wait complete. 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f0281e05700 1 Processor -- start 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f0281e05700 1 -- start start 2024-01-27T23:26:26.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f0281e05700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f0281e05700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f027c106c20 con 0x7f027c0f5410 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f027b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f027b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53924/0 (socket says 172.21.15.194:53924) 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f027b7fe700 1 -- 172.21.15.194:0/2471756281 learned_addr learned my addr 172.21.15.194:0/2471756281 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.592+0000 7f027b7fe700 1 -- 172.21.15.194:0/2471756281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0264005040 con 0x7f027c0f5410 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.593+0000 7f027b7fe700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c106730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0264013510 tx=0x7f026400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.593+0000 7f0273fff700 1 -- 172.21.15.194:0/2471756281 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0264009320 con 0x7f027c0f5410 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.593+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f027c106e20 con 0x7f027c0f5410 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.593+0000 7f0273fff700 1 -- 172.21.15.194:0/2471756281 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0264013f00 con 0x7f027c0f5410 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.593+0000 7f0273fff700 1 -- 172.21.15.194:0/2471756281 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0264014d90 con 0x7f027c0f5410 2024-01-27T23:26:26.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.593+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f027c1072b0 con 0x7f027c0f5410 2024-01-27T23:26:26.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.594+0000 7f0273fff700 1 -- 172.21.15.194:0/2471756281 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02640263a0 con 0x7f027c0f5410 2024-01-27T23:26:26.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.594+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f027c045ad0 con 0x7f027c0f5410 2024-01-27T23:26:26.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.594+0000 7f0273fff700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f026802ff60 0x7f0268032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:26.585 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.594+0000 7f0273fff700 1 -- 172.21.15.194:0/2471756281 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02640412c0 con 0x7f027c0f5410 2024-01-27T23:26:26.585 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.594+0000 7f027affd700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f026802ff60 0x7f0268032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:26.585 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.594+0000 7f027affd700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f026802ff60 0x7f0268032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f026c0096a0 tx=0x7f026c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:26.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:25.266445+0000 mgr.smithi121.hcppvz (mgr.14162) 35 : cluster [DBG] pgmap v28: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:26.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.598+0000 7f0273fff700 1 -- 172.21.15.194:0/2471756281 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0264014600 con 0x7f027c0f5410 2024-01-27T23:26:26.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.806+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f027c0fae60 con 0x7f027c0f5410 2024-01-27T23:26:26.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.806+0000 7f0273fff700 1 -- 172.21.15.194:0/2471756281 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0264014420 con 0x7f027c0f5410 2024-01-27T23:26:26.795 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:26.796 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:26.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f026802ff60 msgr2=0x7f0268032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:26.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f026802ff60 0x7f0268032420 secure :-1 s=READY pgs=24 cs=0 l=1 rev1=1 rx=0x7f026c0096a0 tx=0x7f026c006b40).stop 2024-01-27T23:26:26.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 msgr2=0x7f027c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:26.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c106730 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f0264013510 tx=0x7f026400b040).stop 2024-01-27T23:26:26.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 shutdown_connections 2024-01-27T23:26:26.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f026802ff60 0x7f0268032420 unknown :-1 s=CLOSED pgs=24 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:26.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 --2- 172.21.15.194:0/2471756281 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f027c0f5410 0x7f027c106730 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:26.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 >> 172.21.15.194:0/2471756281 conn(0x7f027c0f0b70 msgr2=0x7f027c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:26.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 shutdown_connections 2024-01-27T23:26:26.798 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:26.808+0000 7f0281e05700 1 -- 172.21.15.194:0/2471756281 wait complete. 2024-01-27T23:26:26.798 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:27.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:26.807578+0000 mon.smithi121 (mon.0) 133 : audit [DBG] from='client.? 172.21.15.194:0/2471756281' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:28.209 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:28.209 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:28.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:27.266818+0000 mgr.smithi121.hcppvz (mgr.14162) 36 : cluster [DBG] pgmap v29: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.351+0000 7f8606c37700 1 -- 172.21.15.194:0/2361439002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 msgr2=0x7f86000f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.351+0000 7f8606c37700 1 --2- 172.21.15.194:0/2361439002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f86000f5830 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f85f4009fb0 tx=0x7f85f4009b30).stop 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.351+0000 7f8606c37700 1 -- 172.21.15.194:0/2361439002 shutdown_connections 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.351+0000 7f8606c37700 1 --2- 172.21.15.194:0/2361439002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f86000f5830 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.351+0000 7f8606c37700 1 -- 172.21.15.194:0/2361439002 >> 172.21.15.194:0/2361439002 conn(0x7f86000f0b70 msgr2=0x7f86000f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.351+0000 7f8606c37700 1 -- 172.21.15.194:0/2361439002 shutdown_connections 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.351+0000 7f8606c37700 1 -- 172.21.15.194:0/2361439002 wait complete. 2024-01-27T23:26:29.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.352+0000 7f8606c37700 1 Processor -- start 2024-01-27T23:26:29.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.352+0000 7f8606c37700 1 -- start start 2024-01-27T23:26:29.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.352+0000 7f8606c37700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f8600105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:29.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.352+0000 7f8606c37700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8600105b20 con 0x7f86000f5410 2024-01-27T23:26:29.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.352+0000 7f86049d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f8600105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.352+0000 7f86049d3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f8600105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53934/0 (socket says 172.21.15.194:53934) 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.352+0000 7f86049d3700 1 -- 172.21.15.194:0/3041901532 learned_addr learned my addr 172.21.15.194:0/3041901532 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.353+0000 7f86049d3700 1 -- 172.21.15.194:0/3041901532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f85f4005040 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.353+0000 7f86049d3700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f8600105630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f85f4013510 tx=0x7f85f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.353+0000 7f85fd7fa700 1 -- 172.21.15.194:0/3041901532 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85f4009320 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.353+0000 7f85fd7fa700 1 -- 172.21.15.194:0/3041901532 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f85f4013f00 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.353+0000 7f85fd7fa700 1 -- 172.21.15.194:0/3041901532 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f85f4014d90 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.353+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8600105d20 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.353+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f86001061b0 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.354+0000 7f85fd7fa700 1 -- 172.21.15.194:0/3041901532 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f85f40263a0 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.354+0000 7f85fd7fa700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f85f002ff60 0x7f85f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.354+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8600045ad0 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.354+0000 7f85fd7fa700 1 -- 172.21.15.194:0/3041901532 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f85f4041080 con 0x7f86000f5410 2024-01-27T23:26:29.344 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.354+0000 7f85fffff700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f85f002ff60 0x7f85f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:29.345 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.355+0000 7f85fffff700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f85f002ff60 0x7f85f0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f85ec0096a0 tx=0x7f85ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:29.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.359+0000 7f85fd7fa700 1 -- 172.21.15.194:0/3041901532 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f85f4012920 con 0x7f86000f5410 2024-01-27T23:26:29.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.564+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f86000faf50 con 0x7f86000f5410 2024-01-27T23:26:29.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.565+0000 7f85fd7fa700 1 -- 172.21.15.194:0/3041901532 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f85f402c090 con 0x7f86000f5410 2024-01-27T23:26:29.554 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:29.554 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:29.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f85f002ff60 msgr2=0x7f85f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:29.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f85f002ff60 0x7f85f0032420 secure :-1 s=READY pgs=25 cs=0 l=1 rev1=1 rx=0x7f85ec0096a0 tx=0x7f85ec006b40).stop 2024-01-27T23:26:29.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 msgr2=0x7f8600105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f8600105630 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7f85f4013510 tx=0x7f85f400b040).stop 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 shutdown_connections 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f85f002ff60 0x7f85f0032420 unknown :-1 s=CLOSED pgs=25 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 --2- 172.21.15.194:0/3041901532 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f86000f5410 0x7f8600105630 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 >> 172.21.15.194:0/3041901532 conn(0x7f86000f0b70 msgr2=0x7f86000f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 shutdown_connections 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:29.567+0000 7f8606c37700 1 -- 172.21.15.194:0/3041901532 wait complete. 2024-01-27T23:26:29.557 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:30.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:29.267139+0000 mgr.smithi121.hcppvz (mgr.14162) 37 : cluster [DBG] pgmap v30: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:30.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:29.566144+0000 mon.smithi121 (mon.0) 134 : audit [DBG] from='client.? 172.21.15.194:0/3041901532' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:30.932 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:30.933 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:32.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.108+0000 7fe675a3e700 1 -- 172.21.15.194:0/1081103845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 msgr2=0x7fe6700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:32.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.108+0000 7fe675a3e700 1 --2- 172.21.15.194:0/1081103845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe6700f5830 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7fe664009fb0 tx=0x7fe664009b30).stop 2024-01-27T23:26:32.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.108+0000 7fe675a3e700 1 -- 172.21.15.194:0/1081103845 shutdown_connections 2024-01-27T23:26:32.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.108+0000 7fe675a3e700 1 --2- 172.21.15.194:0/1081103845 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe6700f5830 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:32.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.108+0000 7fe675a3e700 1 -- 172.21.15.194:0/1081103845 >> 172.21.15.194:0/1081103845 conn(0x7fe6700f0b70 msgr2=0x7fe6700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:32.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.108+0000 7fe675a3e700 1 -- 172.21.15.194:0/1081103845 shutdown_connections 2024-01-27T23:26:32.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.108+0000 7fe675a3e700 1 -- 172.21.15.194:0/1081103845 wait complete. 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.109+0000 7fe675a3e700 1 Processor -- start 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.109+0000 7fe675a3e700 1 -- start start 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe675a3e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe670106710 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe675a3e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe670106c00 con 0x7fe6700f5410 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe66effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe670106710 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe66effd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe670106710 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53958/0 (socket says 172.21.15.194:53958) 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe66effd700 1 -- 172.21.15.194:0/1275352603 learned_addr learned my addr 172.21.15.194:0/1275352603 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe66effd700 1 -- 172.21.15.194:0/1275352603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe664005040 con 0x7fe6700f5410 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe66effd700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe670106710 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe664014000 tx=0x7fe66400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:32.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe6577fe700 1 -- 172.21.15.194:0/1275352603 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6640095f0 con 0x7fe6700f5410 2024-01-27T23:26:32.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe6577fe700 1 -- 172.21.15.194:0/1275352603 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe664007e70 con 0x7fe6700f5410 2024-01-27T23:26:32.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe670106e00 con 0x7fe6700f5410 2024-01-27T23:26:32.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe6577fe700 1 -- 172.21.15.194:0/1275352603 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe664014ea0 con 0x7fe6700f5410 2024-01-27T23:26:32.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.110+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe670107290 con 0x7fe6700f5410 2024-01-27T23:26:32.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.111+0000 7fe6577fe700 1 -- 172.21.15.194:0/1275352603 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe664025070 con 0x7fe6700f5410 2024-01-27T23:26:32.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.111+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe670045ad0 con 0x7fe6700f5410 2024-01-27T23:26:32.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.111+0000 7fe6577fe700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe65802ff60 0x7fe658032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:32.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.111+0000 7fe6577fe700 1 -- 172.21.15.194:0/1275352603 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe664040c90 con 0x7fe6700f5410 2024-01-27T23:26:32.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.112+0000 7fe66e7fc700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe65802ff60 0x7fe658032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:32.102 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.112+0000 7fe66e7fc700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe65802ff60 0x7fe658032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe6600096a0 tx=0x7fe660006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:32.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.116+0000 7fe6577fe700 1 -- 172.21.15.194:0/1275352603 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe66400f710 con 0x7fe6700f5410 2024-01-27T23:26:32.312 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.322+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe6700fae60 con 0x7fe6700f5410 2024-01-27T23:26:32.312 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.323+0000 7fe6577fe700 1 -- 172.21.15.194:0/1275352603 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe664012330 con 0x7fe6700f5410 2024-01-27T23:26:32.312 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:32.312 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe65802ff60 msgr2=0x7fe658032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe65802ff60 0x7fe658032420 secure :-1 s=READY pgs=26 cs=0 l=1 rev1=1 rx=0x7fe6600096a0 tx=0x7fe660006b40).stop 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 msgr2=0x7fe670106710 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe670106710 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fe664014000 tx=0x7fe66400b040).stop 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 shutdown_connections 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe65802ff60 0x7fe658032420 unknown :-1 s=CLOSED pgs=26 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 --2- 172.21.15.194:0/1275352603 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6700f5410 0x7fe670106710 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 >> 172.21.15.194:0/1275352603 conn(0x7fe6700f0b70 msgr2=0x7fe67005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 shutdown_connections 2024-01-27T23:26:32.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:32.324+0000 7fe675a3e700 1 -- 172.21.15.194:0/1275352603 wait complete. 2024-01-27T23:26:32.315 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:32.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:31.267414+0000 mgr.smithi121.hcppvz (mgr.14162) 38 : cluster [DBG] pgmap v31: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:33.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:33 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:32.324125+0000 mon.smithi121 (mon.0) 135 : audit [DBG] from='client.? 172.21.15.194:0/1275352603' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:33.709 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:33.710 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:34.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:33.267650+0000 mgr.smithi121.hcppvz (mgr.14162) 39 : cluster [DBG] pgmap v32: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.919+0000 7ff848546700 1 -- 172.21.15.194:0/1576581085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 msgr2=0x7ff8400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.919+0000 7ff848546700 1 --2- 172.21.15.194:0/1576581085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff8400f5830 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7ff830009fb0 tx=0x7ff830009b30).stop 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.919+0000 7ff848546700 1 -- 172.21.15.194:0/1576581085 shutdown_connections 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.919+0000 7ff848546700 1 --2- 172.21.15.194:0/1576581085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff8400f5830 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.919+0000 7ff848546700 1 -- 172.21.15.194:0/1576581085 >> 172.21.15.194:0/1576581085 conn(0x7ff8400f0b70 msgr2=0x7ff8400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.920+0000 7ff848546700 1 -- 172.21.15.194:0/1576581085 shutdown_connections 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.920+0000 7ff848546700 1 -- 172.21.15.194:0/1576581085 wait complete. 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.920+0000 7ff848546700 1 Processor -- start 2024-01-27T23:26:34.911 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.920+0000 7ff848546700 1 -- start start 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.920+0000 7ff848546700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff840109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.920+0000 7ff848546700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff840060220 con 0x7ff8400f5410 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.920+0000 7ff8462e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff840109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff8462e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff840109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53964/0 (socket says 172.21.15.194:53964) 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff8462e2700 1 -- 172.21.15.194:0/2500504487 learned_addr learned my addr 172.21.15.194:0/2500504487 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff8462e2700 1 -- 172.21.15.194:0/2500504487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff830005040 con 0x7ff8400f5410 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff8462e2700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff840109420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7ff830013510 tx=0x7ff83000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff836ffd700 1 -- 172.21.15.194:0/2500504487 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff830009320 con 0x7ff8400f5410 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff840060420 con 0x7ff8400f5410 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff836ffd700 1 -- 172.21.15.194:0/2500504487 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff830013f00 con 0x7ff8400f5410 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff836ffd700 1 -- 172.21.15.194:0/2500504487 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff830014d90 con 0x7ff8400f5410 2024-01-27T23:26:34.912 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.921+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff840109ad0 con 0x7ff8400f5410 2024-01-27T23:26:34.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.922+0000 7ff836ffd700 1 -- 172.21.15.194:0/2500504487 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff830014420 con 0x7ff8400f5410 2024-01-27T23:26:34.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.922+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff840059100 con 0x7ff8400f5410 2024-01-27T23:26:34.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.922+0000 7ff836ffd700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff82c02ff60 0x7ff82c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:34.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.922+0000 7ff836ffd700 1 -- 172.21.15.194:0/2500504487 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8300410c0 con 0x7ff8400f5410 2024-01-27T23:26:34.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.924+0000 7ff845ae1700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff82c02ff60 0x7ff82c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:34.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.924+0000 7ff845ae1700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff82c02ff60 0x7ff82c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff83c0096a0 tx=0x7ff83c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:34.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:34.927+0000 7ff836ffd700 1 -- 172.21.15.194:0/2500504487 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8300095f0 con 0x7ff8400f5410 2024-01-27T23:26:35.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.131+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff840045ad0 con 0x7ff8400f5410 2024-01-27T23:26:35.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.132+0000 7ff836ffd700 1 -- 172.21.15.194:0/2500504487 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff83001db70 con 0x7ff8400f5410 2024-01-27T23:26:35.121 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:35.121 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff82c02ff60 msgr2=0x7ff82c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff82c02ff60 0x7ff82c032420 secure :-1 s=READY pgs=27 cs=0 l=1 rev1=1 rx=0x7ff83c0096a0 tx=0x7ff83c006b40).stop 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 msgr2=0x7ff840109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff840109420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7ff830013510 tx=0x7ff83000b040).stop 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 shutdown_connections 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff82c02ff60 0x7ff82c032420 unknown :-1 s=CLOSED pgs=27 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 --2- 172.21.15.194:0/2500504487 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8400f5410 0x7ff840109420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 >> 172.21.15.194:0/2500504487 conn(0x7ff8400f0b70 msgr2=0x7ff8400f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 shutdown_connections 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:35.133+0000 7ff848546700 1 -- 172.21.15.194:0/2500504487 wait complete. 2024-01-27T23:26:35.123 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:35.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:35 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:35.133014+0000 mon.smithi121 (mon.0) 136 : audit [DBG] from='client.? 172.21.15.194:0/2500504487' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:36.528 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:36.528 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:36.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:35.267994+0000 mgr.smithi121.hcppvz (mgr.14162) 40 : cluster [DBG] pgmap v33: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.693+0000 7fbe2c814700 1 -- 172.21.15.194:0/1815469637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 msgr2=0x7fbe280f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.693+0000 7fbe2c814700 1 --2- 172.21.15.194:0/1815469637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe280f6930 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fbe1c009fb0 tx=0x7fbe1c009b30).stop 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.693+0000 7fbe2c814700 1 -- 172.21.15.194:0/1815469637 shutdown_connections 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.693+0000 7fbe2c814700 1 --2- 172.21.15.194:0/1815469637 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe280f6930 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.693+0000 7fbe2c814700 1 -- 172.21.15.194:0/1815469637 >> 172.21.15.194:0/1815469637 conn(0x7fbe280f1c70 msgr2=0x7fbe280f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.693+0000 7fbe2c814700 1 -- 172.21.15.194:0/1815469637 shutdown_connections 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.693+0000 7fbe2c814700 1 -- 172.21.15.194:0/1815469637 wait complete. 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2c814700 1 Processor -- start 2024-01-27T23:26:37.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2c814700 1 -- start start 2024-01-27T23:26:37.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2c814700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe28106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:37.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2c814700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbe28106c20 con 0x7fbe280f6510 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe28106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe28106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45010/0 (socket says 172.21.15.194:45010) 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2659c700 1 -- 172.21.15.194:0/3723455445 learned_addr learned my addr 172.21.15.194:0/3723455445 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2659c700 1 -- 172.21.15.194:0/3723455445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbe1c005040 con 0x7fbe280f6510 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.694+0000 7fbe2659c700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe28106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fbe1c013510 tx=0x7fbe1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.695+0000 7fbe16ffd700 1 -- 172.21.15.194:0/3723455445 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe1c009320 con 0x7fbe280f6510 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.695+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbe28106e20 con 0x7fbe280f6510 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.695+0000 7fbe16ffd700 1 -- 172.21.15.194:0/3723455445 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbe1c013f00 con 0x7fbe280f6510 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.695+0000 7fbe16ffd700 1 -- 172.21.15.194:0/3723455445 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbe1c014d90 con 0x7fbe280f6510 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.695+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbe281072b0 con 0x7fbe280f6510 2024-01-27T23:26:37.686 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.695+0000 7fbe16ffd700 1 -- 172.21.15.194:0/3723455445 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbe1c014420 con 0x7fbe280f6510 2024-01-27T23:26:37.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.696+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbe28045ad0 con 0x7fbe280f6510 2024-01-27T23:26:37.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.696+0000 7fbe16ffd700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbe1002ff60 0x7fbe10032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:37.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.696+0000 7fbe16ffd700 1 -- 172.21.15.194:0/3723455445 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbe1c0410c0 con 0x7fbe280f6510 2024-01-27T23:26:37.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.696+0000 7fbe25d9b700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbe1002ff60 0x7fbe10032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:37.687 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.696+0000 7fbe25d9b700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbe1002ff60 0x7fbe10032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fbe180096a0 tx=0x7fbe18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:37.689 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.700+0000 7fbe16ffd700 1 -- 172.21.15.194:0/3723455445 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbe1c0095f0 con 0x7fbe280f6510 2024-01-27T23:26:37.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.906+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbe280fbf60 con 0x7fbe280f6510 2024-01-27T23:26:37.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.907+0000 7fbe16ffd700 1 -- 172.21.15.194:0/3723455445 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbe1c01db70 con 0x7fbe280f6510 2024-01-27T23:26:37.896 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:37.896 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:37.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbe1002ff60 msgr2=0x7fbe10032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:37.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbe1002ff60 0x7fbe10032420 secure :-1 s=READY pgs=28 cs=0 l=1 rev1=1 rx=0x7fbe180096a0 tx=0x7fbe18006b40).stop 2024-01-27T23:26:37.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 msgr2=0x7fbe28106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe28106730 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7fbe1c013510 tx=0x7fbe1c00b040).stop 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 shutdown_connections 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbe1002ff60 0x7fbe10032420 unknown :-1 s=CLOSED pgs=28 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 --2- 172.21.15.194:0/3723455445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbe280f6510 0x7fbe28106730 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 >> 172.21.15.194:0/3723455445 conn(0x7fbe280f1c70 msgr2=0x7fbe280f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 shutdown_connections 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:37.909+0000 7fbe2c814700 1 -- 172.21.15.194:0/3723455445 wait complete. 2024-01-27T23:26:37.899 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:38.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:37.268273+0000 mgr.smithi121.hcppvz (mgr.14162) 41 : cluster [DBG] pgmap v34: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:38.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:37.908450+0000 mon.smithi121 (mon.0) 137 : audit [DBG] from='client.? 172.21.15.194:0/3723455445' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:39.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:39.277 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:40.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 -- 172.21.15.194:0/3460872215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 msgr2=0x7ffb8c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 --2- 172.21.15.194:0/3460872215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c0f5830 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7ffb88009fb0 tx=0x7ffb88009b30).stop 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 -- 172.21.15.194:0/3460872215 shutdown_connections 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 --2- 172.21.15.194:0/3460872215 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c0f5830 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 -- 172.21.15.194:0/3460872215 >> 172.21.15.194:0/3460872215 conn(0x7ffb8c0f0b70 msgr2=0x7ffb8c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 -- 172.21.15.194:0/3460872215 shutdown_connections 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 -- 172.21.15.194:0/3460872215 wait complete. 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 Processor -- start 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 -- start start 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.423+0000 7ffb93382700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ffb8c060240 con 0x7ffb8c0f5410 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb9111e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb9111e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45024/0 (socket says 172.21.15.194:45024) 2024-01-27T23:26:40.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb9111e700 1 -- 172.21.15.194:0/910067596 learned_addr learned my addr 172.21.15.194:0/910067596 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:40.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb9111e700 1 -- 172.21.15.194:0/910067596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ffb88005040 con 0x7ffb8c0f5410 2024-01-27T23:26:40.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb9111e700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c109430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ffb88013510 tx=0x7ffb8800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:40.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb81ffb700 1 -- 172.21.15.194:0/910067596 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb88009320 con 0x7ffb8c0f5410 2024-01-27T23:26:40.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ffb8c060440 con 0x7ffb8c0f5410 2024-01-27T23:26:40.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb81ffb700 1 -- 172.21.15.194:0/910067596 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ffb88013f00 con 0x7ffb8c0f5410 2024-01-27T23:26:40.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb81ffb700 1 -- 172.21.15.194:0/910067596 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ffb88014d90 con 0x7ffb8c0f5410 2024-01-27T23:26:40.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.424+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ffb8c109ae0 con 0x7ffb8c0f5410 2024-01-27T23:26:40.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.425+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ffb8c045ad0 con 0x7ffb8c0f5410 2024-01-27T23:26:40.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.425+0000 7ffb81ffb700 1 -- 172.21.15.194:0/910067596 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ffb880095f0 con 0x7ffb8c0f5410 2024-01-27T23:26:40.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.425+0000 7ffb81ffb700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ffb7802ff60 0x7ffb78032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:40.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.425+0000 7ffb81ffb700 1 -- 172.21.15.194:0/910067596 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ffb88041080 con 0x7ffb8c0f5410 2024-01-27T23:26:40.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.426+0000 7ffb9091d700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ffb7802ff60 0x7ffb78032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:40.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.426+0000 7ffb9091d700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ffb7802ff60 0x7ffb78032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ffb7c0096a0 tx=0x7ffb7c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:40.419 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.430+0000 7ffb81ffb700 1 -- 172.21.15.194:0/910067596 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ffb88014420 con 0x7ffb8c0f5410 2024-01-27T23:26:40.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:39.268545+0000 mgr.smithi121.hcppvz (mgr.14162) 42 : cluster [DBG] pgmap v35: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:40.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.634+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ffb8c0fae60 con 0x7ffb8c0f5410 2024-01-27T23:26:40.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.635+0000 7ffb81ffb700 1 -- 172.21.15.194:0/910067596 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ffb88010230 con 0x7ffb8c0f5410 2024-01-27T23:26:40.624 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:40.624 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:40.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ffb7802ff60 msgr2=0x7ffb78032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:40.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ffb7802ff60 0x7ffb78032420 secure :-1 s=READY pgs=29 cs=0 l=1 rev1=1 rx=0x7ffb7c0096a0 tx=0x7ffb7c006b40).stop 2024-01-27T23:26:40.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 msgr2=0x7ffb8c109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:40.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c109430 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7ffb88013510 tx=0x7ffb8800b040).stop 2024-01-27T23:26:40.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 shutdown_connections 2024-01-27T23:26:40.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ffb7802ff60 0x7ffb78032420 unknown :-1 s=CLOSED pgs=29 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:40.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 --2- 172.21.15.194:0/910067596 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ffb8c0f5410 0x7ffb8c109430 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:40.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 >> 172.21.15.194:0/910067596 conn(0x7ffb8c0f0b70 msgr2=0x7ffb8c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:40.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 shutdown_connections 2024-01-27T23:26:40.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:40.637+0000 7ffb93382700 1 -- 172.21.15.194:0/910067596 wait complete. 2024-01-27T23:26:40.627 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:41.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:41 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:40.636320+0000 mon.smithi121 (mon.0) 138 : audit [DBG] from='client.? 172.21.15.194:0/910067596' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:41.963 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:41.963 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:42.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:41.268797+0000 mgr.smithi121.hcppvz (mgr.14162) 43 : cluster [DBG] pgmap v36: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:44.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 -- 172.21.15.194:0/2165723213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 msgr2=0x7fcd5c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 --2- 172.21.15.194:0/2165723213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c0f5830 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcd4c009fb0 tx=0x7fcd4c009b30).stop 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 -- 172.21.15.194:0/2165723213 shutdown_connections 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 --2- 172.21.15.194:0/2165723213 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c0f5830 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 -- 172.21.15.194:0/2165723213 >> 172.21.15.194:0/2165723213 conn(0x7fcd5c0f0b70 msgr2=0x7fcd5c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 -- 172.21.15.194:0/2165723213 shutdown_connections 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 -- 172.21.15.194:0/2165723213 wait complete. 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 Processor -- start 2024-01-27T23:26:44.278 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.288+0000 7fcd632cc700 1 -- start start 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.289+0000 7fcd632cc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.289+0000 7fcd632cc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcd5c105b20 con 0x7fcd5c0f5410 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.289+0000 7fcd61068700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.289+0000 7fcd61068700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45042/0 (socket says 172.21.15.194:45042) 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.289+0000 7fcd61068700 1 -- 172.21.15.194:0/3983346636 learned_addr learned my addr 172.21.15.194:0/3983346636 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.289+0000 7fcd61068700 1 -- 172.21.15.194:0/3983346636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcd4c005040 con 0x7fcd5c0f5410 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.289+0000 7fcd61068700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c105630 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcd4c013510 tx=0x7fcd4c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:44.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.290+0000 7fcd51ffb700 1 -- 172.21.15.194:0/3983346636 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd4c009320 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.290+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcd5c105d20 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.290+0000 7fcd51ffb700 1 -- 172.21.15.194:0/3983346636 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcd4c013f00 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.290+0000 7fcd51ffb700 1 -- 172.21.15.194:0/3983346636 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcd4c014d90 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.290+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcd5c1061b0 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.291+0000 7fcd51ffb700 1 -- 172.21.15.194:0/3983346636 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcd4c0095f0 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.291+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcd5c045ad0 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.291+0000 7fcd51ffb700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcd4802ff60 0x7fcd48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.291+0000 7fcd51ffb700 1 -- 172.21.15.194:0/3983346636 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcd4c041000 con 0x7fcd5c0f5410 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.291+0000 7fcd60867700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcd4802ff60 0x7fcd48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:44.281 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.292+0000 7fcd60867700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcd4802ff60 0x7fcd48032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcd580096a0 tx=0x7fcd58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:44.284 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.296+0000 7fcd51ffb700 1 -- 172.21.15.194:0/3983346636 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcd4c01d7f0 con 0x7fcd5c0f5410 2024-01-27T23:26:44.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.499+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcd5c0fae60 con 0x7fcd5c0f5410 2024-01-27T23:26:44.489 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.500+0000 7fcd51ffb700 1 -- 172.21.15.194:0/3983346636 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcd4c010230 con 0x7fcd5c0f5410 2024-01-27T23:26:44.490 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:44.490 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcd4802ff60 msgr2=0x7fcd48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcd4802ff60 0x7fcd48032420 secure :-1 s=READY pgs=30 cs=0 l=1 rev1=1 rx=0x7fcd580096a0 tx=0x7fcd58006b40).stop 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 msgr2=0x7fcd5c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c105630 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fcd4c013510 tx=0x7fcd4c00b040).stop 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 shutdown_connections 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcd4802ff60 0x7fcd48032420 unknown :-1 s=CLOSED pgs=30 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 --2- 172.21.15.194:0/3983346636 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcd5c0f5410 0x7fcd5c105630 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 >> 172.21.15.194:0/3983346636 conn(0x7fcd5c0f0b70 msgr2=0x7fcd5c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 shutdown_connections 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:44.502+0000 7fcd632cc700 1 -- 172.21.15.194:0/3983346636 wait complete. 2024-01-27T23:26:44.492 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:44.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:43.269099+0000 mgr.smithi121.hcppvz (mgr.14162) 44 : cluster [DBG] pgmap v37: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:45.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:45 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:44.501512+0000 mon.smithi121 (mon.0) 139 : audit [DBG] from='client.? 172.21.15.194:0/3983346636' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:45.874 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:45.875 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:46.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:45.269443+0000 mgr.smithi121.hcppvz (mgr.14162) 45 : cluster [DBG] pgmap v38: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:47.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 -- 172.21.15.194:0/3258664598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 msgr2=0x7f48ac0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:47.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 --2- 172.21.15.194:0/3258664598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac0f5830 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f489c009fb0 tx=0x7f489c009b30).stop 2024-01-27T23:26:47.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 -- 172.21.15.194:0/3258664598 shutdown_connections 2024-01-27T23:26:47.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 --2- 172.21.15.194:0/3258664598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac0f5830 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:47.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 -- 172.21.15.194:0/3258664598 >> 172.21.15.194:0/3258664598 conn(0x7f48ac0f0b70 msgr2=0x7f48ac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:47.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 -- 172.21.15.194:0/3258664598 shutdown_connections 2024-01-27T23:26:47.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 -- 172.21.15.194:0/3258664598 wait complete. 2024-01-27T23:26:47.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.059+0000 7f48b2631700 1 Processor -- start 2024-01-27T23:26:47.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.060+0000 7f48b2631700 1 -- start start 2024-01-27T23:26:47.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.060+0000 7f48b2631700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:47.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.060+0000 7f48b2631700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f48ac106c10 con 0x7f48ac0f5410 2024-01-27T23:26:47.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.060+0000 7f48abfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.060+0000 7f48abfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:47752/0 (socket says 172.21.15.194:47752) 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.060+0000 7f48abfff700 1 -- 172.21.15.194:0/2546228173 learned_addr learned my addr 172.21.15.194:0/2546228173 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.061+0000 7f48abfff700 1 -- 172.21.15.194:0/2546228173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f489c005040 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.061+0000 7f48abfff700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac106720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f489c014000 tx=0x7f489c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.061+0000 7f48a8ff9700 1 -- 172.21.15.194:0/2546228173 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f489c0095f0 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.061+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f48ac106e10 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.061+0000 7f48a8ff9700 1 -- 172.21.15.194:0/2546228173 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f489c007e70 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.061+0000 7f48a8ff9700 1 -- 172.21.15.194:0/2546228173 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f489c014ea0 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.061+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f48ac1072a0 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.062+0000 7f48a8ff9700 1 -- 172.21.15.194:0/2546228173 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f489c025070 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.062+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f48ac045ad0 con 0x7f48ac0f5410 2024-01-27T23:26:47.052 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.062+0000 7f48a8ff9700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f489402ff60 0x7f4894032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:47.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.062+0000 7f48a8ff9700 1 -- 172.21.15.194:0/2546228173 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f489c0409a0 con 0x7f48ac0f5410 2024-01-27T23:26:47.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.062+0000 7f48ab7fe700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f489402ff60 0x7f4894032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:47.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.063+0000 7f48ab7fe700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f489402ff60 0x7f4894032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:47.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.067+0000 7f48a8ff9700 1 -- 172.21.15.194:0/2546228173 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f489c026350 con 0x7f48ac0f5410 2024-01-27T23:26:47.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.276+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f48ac108150 con 0x7f48ac0f5410 2024-01-27T23:26:47.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.276+0000 7f48a8ff9700 1 -- 172.21.15.194:0/2546228173 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f489c012330 con 0x7f48ac0f5410 2024-01-27T23:26:47.266 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:47.266 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:47.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f489402ff60 msgr2=0x7f4894032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:47.267 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f489402ff60 0x7f4894032420 secure :-1 s=READY pgs=31 cs=0 l=1 rev1=1 rx=0x7f48a00096a0 tx=0x7f48a0006b40).stop 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 msgr2=0x7f48ac106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac106720 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f489c014000 tx=0x7f489c00b040).stop 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 shutdown_connections 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f489402ff60 0x7f4894032420 unknown :-1 s=CLOSED pgs=31 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 --2- 172.21.15.194:0/2546228173 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f48ac0f5410 0x7f48ac106720 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 >> 172.21.15.194:0/2546228173 conn(0x7f48ac0f0b70 msgr2=0x7f48ac05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 shutdown_connections 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:47.278+0000 7f48b2631700 1 -- 172.21.15.194:0/2546228173 wait complete. 2024-01-27T23:26:47.268 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:47.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:47 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:47.277705+0000 mon.smithi121 (mon.0) 140 : audit [DBG] from='client.? 172.21.15.194:0/2546228173' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:48.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:47.269705+0000 mgr.smithi121.hcppvz (mgr.14162) 46 : cluster [DBG] pgmap v39: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:48.644 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:48.645 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:49.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 -- 172.21.15.194:0/272002151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 msgr2=0x7f4b740f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:49.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 --2- 172.21.15.194:0/272002151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b740f55c0 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7f4b68009fb0 tx=0x7f4b68009b30).stop 2024-01-27T23:26:49.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 -- 172.21.15.194:0/272002151 shutdown_connections 2024-01-27T23:26:49.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 --2- 172.21.15.194:0/272002151 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b740f55c0 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:49.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 -- 172.21.15.194:0/272002151 >> 172.21.15.194:0/272002151 conn(0x7f4b740f09a0 msgr2=0x7f4b740f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:49.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 -- 172.21.15.194:0/272002151 shutdown_connections 2024-01-27T23:26:49.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 -- 172.21.15.194:0/272002151 wait complete. 2024-01-27T23:26:49.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 Processor -- start 2024-01-27T23:26:49.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 -- start start 2024-01-27T23:26:49.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.867+0000 7f4b79a8d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b741091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:49.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b79a8d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4b74060240 con 0x7f4b740f51a0 2024-01-27T23:26:49.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b78a8b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b741091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:49.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b78a8b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b741091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:47764/0 (socket says 172.21.15.194:47764) 2024-01-27T23:26:49.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b78a8b700 1 -- 172.21.15.194:0/2407818563 learned_addr learned my addr 172.21.15.194:0/2407818563 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:49.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b78a8b700 1 -- 172.21.15.194:0/2407818563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4b68005040 con 0x7f4b740f51a0 2024-01-27T23:26:49.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b78a8b700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b741091b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4b68013510 tx=0x7f4b6800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b717fa700 1 -- 172.21.15.194:0/2407818563 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b68009320 con 0x7f4b740f51a0 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b717fa700 1 -- 172.21.15.194:0/2407818563 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4b68013f00 con 0x7f4b740f51a0 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4b74060440 con 0x7f4b740f51a0 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b717fa700 1 -- 172.21.15.194:0/2407818563 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4b68014dc0 con 0x7f4b740f51a0 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.868+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4b74109860 con 0x7f4b740f51a0 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.869+0000 7f4b717fa700 1 -- 172.21.15.194:0/2407818563 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4b6801d3d0 con 0x7f4b740f51a0 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.869+0000 7f4b717fa700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4b6402ff60 0x7f4b64032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:49.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.869+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4b74045ad0 con 0x7f4b740f51a0 2024-01-27T23:26:49.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.869+0000 7f4b717fa700 1 -- 172.21.15.194:0/2407818563 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4b68041330 con 0x7f4b740f51a0 2024-01-27T23:26:49.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.869+0000 7f4b73fff700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4b6402ff60 0x7f4b64032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:49.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.870+0000 7f4b73fff700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4b6402ff60 0x7f4b64032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4b600096a0 tx=0x7f4b60006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:49.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:49.874+0000 7f4b717fa700 1 -- 172.21.15.194:0/2407818563 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4b68012330 con 0x7f4b740f51a0 2024-01-27T23:26:50.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.078+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4b7410a080 con 0x7f4b740f51a0 2024-01-27T23:26:50.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.079+0000 7f4b717fa700 1 -- 172.21.15.194:0/2407818563 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4b68014420 con 0x7f4b740f51a0 2024-01-27T23:26:50.068 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:50.068 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:50.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4b6402ff60 msgr2=0x7f4b64032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4b6402ff60 0x7f4b64032420 secure :-1 s=READY pgs=32 cs=0 l=1 rev1=1 rx=0x7f4b600096a0 tx=0x7f4b60006b40).stop 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 msgr2=0x7f4b741091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b741091b0 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7f4b68013510 tx=0x7f4b6800b040).stop 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 shutdown_connections 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4b6402ff60 0x7f4b64032420 unknown :-1 s=CLOSED pgs=32 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 --2- 172.21.15.194:0/2407818563 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4b740f51a0 0x7f4b741091b0 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 >> 172.21.15.194:0/2407818563 conn(0x7f4b740f09a0 msgr2=0x7f4b740f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 shutdown_connections 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:50.080+0000 7f4b79a8d700 1 -- 172.21.15.194:0/2407818563 wait complete. 2024-01-27T23:26:50.070 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:50.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:49.270063+0000 mgr.smithi121.hcppvz (mgr.14162) 47 : cluster [DBG] pgmap v40: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:50.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:50.080035+0000 mon.smithi121 (mon.0) 141 : audit [DBG] from='client.? 172.21.15.194:0/2407818563' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:51.495 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:51.495 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:52.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:51.270378+0000 mgr.smithi121.hcppvz (mgr.14162) 48 : cluster [DBG] pgmap v41: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:52.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 -- 172.21.15.194:0/711748838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 msgr2=0x7f15b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:52.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 --2- 172.21.15.194:0/711748838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b40f6930 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f15a4009fb0 tx=0x7f15a4009b30).stop 2024-01-27T23:26:52.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 -- 172.21.15.194:0/711748838 shutdown_connections 2024-01-27T23:26:52.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 --2- 172.21.15.194:0/711748838 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b40f6930 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:52.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 -- 172.21.15.194:0/711748838 >> 172.21.15.194:0/711748838 conn(0x7f15b40f1c70 msgr2=0x7f15b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:52.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 -- 172.21.15.194:0/711748838 shutdown_connections 2024-01-27T23:26:52.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 -- 172.21.15.194:0/711748838 wait complete. 2024-01-27T23:26:52.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 Processor -- start 2024-01-27T23:26:52.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.630+0000 7f15bba5a700 1 -- start start 2024-01-27T23:26:52.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15bba5a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:52.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15bba5a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15b4106c20 con 0x7f15b40f6510 2024-01-27T23:26:52.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15b97f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15b97f6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:47778/0 (socket says 172.21.15.194:47778) 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15b97f6700 1 -- 172.21.15.194:0/1490010584 learned_addr learned my addr 172.21.15.194:0/1490010584 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15b97f6700 1 -- 172.21.15.194:0/1490010584 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15a4005040 con 0x7f15b40f6510 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15b97f6700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b4106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f15a4013510 tx=0x7f15a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.631+0000 7f15a9ffb700 1 -- 172.21.15.194:0/1490010584 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15a4009320 con 0x7f15b40f6510 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.632+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15b4106e20 con 0x7f15b40f6510 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.632+0000 7f15a9ffb700 1 -- 172.21.15.194:0/1490010584 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15a4013f00 con 0x7f15b40f6510 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.632+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15b41072b0 con 0x7f15b40f6510 2024-01-27T23:26:52.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.632+0000 7f15a9ffb700 1 -- 172.21.15.194:0/1490010584 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15a4014db0 con 0x7f15b40f6510 2024-01-27T23:26:52.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.632+0000 7f15a9ffb700 1 -- 172.21.15.194:0/1490010584 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15a4014420 con 0x7f15b40f6510 2024-01-27T23:26:52.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.633+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15b4045ad0 con 0x7f15b40f6510 2024-01-27T23:26:52.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.633+0000 7f15a9ffb700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15a002ff60 0x7f15a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:52.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.633+0000 7f15a9ffb700 1 -- 172.21.15.194:0/1490010584 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15a40410a0 con 0x7f15b40f6510 2024-01-27T23:26:52.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.633+0000 7f15b8ff5700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15a002ff60 0x7f15a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:52.623 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.633+0000 7f15b8ff5700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15a002ff60 0x7f15a0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f15b00096a0 tx=0x7f15b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:52.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.637+0000 7f15a9ffb700 1 -- 172.21.15.194:0/1490010584 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15a40095f0 con 0x7f15b40f6510 2024-01-27T23:26:52.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.847+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15b40fbf60 con 0x7f15b40f6510 2024-01-27T23:26:52.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.848+0000 7f15a9ffb700 1 -- 172.21.15.194:0/1490010584 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f15a401cb10 con 0x7f15b40f6510 2024-01-27T23:26:52.837 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:52.837 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:52.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15a002ff60 msgr2=0x7f15a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:52.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15a002ff60 0x7f15a0032420 secure :-1 s=READY pgs=33 cs=0 l=1 rev1=1 rx=0x7f15b00096a0 tx=0x7f15b0006b40).stop 2024-01-27T23:26:52.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 msgr2=0x7f15b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:52.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b4106730 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f15a4013510 tx=0x7f15a400b040).stop 2024-01-27T23:26:52.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 shutdown_connections 2024-01-27T23:26:52.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15a002ff60 0x7f15a0032420 unknown :-1 s=CLOSED pgs=33 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:52.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 --2- 172.21.15.194:0/1490010584 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15b40f6510 0x7f15b4106730 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:52.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 >> 172.21.15.194:0/1490010584 conn(0x7f15b40f1c70 msgr2=0x7f15b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:52.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 shutdown_connections 2024-01-27T23:26:52.839 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:52.849+0000 7f15bba5a700 1 -- 172.21.15.194:0/1490010584 wait complete. 2024-01-27T23:26:52.839 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:53.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:53 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:52.848866+0000 mon.smithi121 (mon.0) 142 : audit [DBG] from='client.? 172.21.15.194:0/1490010584' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:54.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:54.220 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:54.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:53.270608+0000 mgr.smithi121.hcppvz (mgr.14162) 49 : cluster [DBG] pgmap v42: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 -- 172.21.15.194:0/3110483688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 msgr2=0x7fde2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 --2- 172.21.15.194:0/3110483688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c0f5830 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fde20009fb0 tx=0x7fde20009b30).stop 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 -- 172.21.15.194:0/3110483688 shutdown_connections 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 --2- 172.21.15.194:0/3110483688 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c0f5830 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 -- 172.21.15.194:0/3110483688 >> 172.21.15.194:0/3110483688 conn(0x7fde2c0f0b70 msgr2=0x7fde2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 -- 172.21.15.194:0/3110483688 shutdown_connections 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 -- 172.21.15.194:0/3110483688 wait complete. 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.442+0000 7fde32aa0700 1 Processor -- start 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde32aa0700 1 -- start start 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde32aa0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:55.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde32aa0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fde2c106c20 con 0x7fde2c0f5410 2024-01-27T23:26:55.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde3083c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:55.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde3083c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:57336/0 (socket says 172.21.15.194:57336) 2024-01-27T23:26:55.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde3083c700 1 -- 172.21.15.194:0/352080063 learned_addr learned my addr 172.21.15.194:0/352080063 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:55.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde3083c700 1 -- 172.21.15.194:0/352080063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fde20005040 con 0x7fde2c0f5410 2024-01-27T23:26:55.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.443+0000 7fde3083c700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fde20013510 tx=0x7fde2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:55.434 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.444+0000 7fde297fa700 1 -- 172.21.15.194:0/352080063 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde20009320 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.444+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fde2c106e20 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.444+0000 7fde297fa700 1 -- 172.21.15.194:0/352080063 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fde20013f00 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.444+0000 7fde297fa700 1 -- 172.21.15.194:0/352080063 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fde20014d90 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.444+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fde2c1072b0 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.444+0000 7fde297fa700 1 -- 172.21.15.194:0/352080063 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fde20014420 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.444+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fde2c045ad0 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.445+0000 7fde297fa700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fde1c02ff60 0x7fde1c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.445+0000 7fde297fa700 1 -- 172.21.15.194:0/352080063 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fde200410c0 con 0x7fde2c0f5410 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.445+0000 7fde2bfff700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fde1c02ff60 0x7fde1c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:55.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.445+0000 7fde2bfff700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fde1c02ff60 0x7fde1c032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fde180096a0 tx=0x7fde18006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:55.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.449+0000 7fde297fa700 1 -- 172.21.15.194:0/352080063 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fde200095f0 con 0x7fde2c0f5410 2024-01-27T23:26:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.657+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fde2c0fbf90 con 0x7fde2c0f5410 2024-01-27T23:26:55.646 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.657+0000 7fde297fa700 1 -- 172.21.15.194:0/352080063 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fde2001db70 con 0x7fde2c0f5410 2024-01-27T23:26:55.646 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:55.647 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fde1c02ff60 msgr2=0x7fde1c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fde1c02ff60 0x7fde1c032420 secure :-1 s=READY pgs=34 cs=0 l=1 rev1=1 rx=0x7fde180096a0 tx=0x7fde18006b40).stop 2024-01-27T23:26:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 msgr2=0x7fde2c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:55.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c106730 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7fde20013510 tx=0x7fde2000b040).stop 2024-01-27T23:26:55.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 shutdown_connections 2024-01-27T23:26:55.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fde1c02ff60 0x7fde1c032420 unknown :-1 s=CLOSED pgs=34 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:55.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 --2- 172.21.15.194:0/352080063 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fde2c0f5410 0x7fde2c106730 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:55.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 >> 172.21.15.194:0/352080063 conn(0x7fde2c0f0b70 msgr2=0x7fde2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:55.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 shutdown_connections 2024-01-27T23:26:55.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:55.659+0000 7fde32aa0700 1 -- 172.21.15.194:0/352080063 wait complete. 2024-01-27T23:26:55.649 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:56.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:55.270845+0000 mgr.smithi121.hcppvz (mgr.14162) 50 : cluster [DBG] pgmap v43: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:56.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:55.658632+0000 mon.smithi121 (mon.0) 143 : audit [DBG] from='client.? 172.21.15.194:0/352080063' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:57.025 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:57.025 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:26:58.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.179+0000 7f416bbc2700 1 -- 172.21.15.194:0/3060864242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 msgr2=0x7f41640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:58.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.179+0000 7f416bbc2700 1 --2- 172.21.15.194:0/3060864242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f41640f5830 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f4158009fb0 tx=0x7f4158009b30).stop 2024-01-27T23:26:58.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.179+0000 7f416bbc2700 1 -- 172.21.15.194:0/3060864242 shutdown_connections 2024-01-27T23:26:58.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.179+0000 7f416bbc2700 1 --2- 172.21.15.194:0/3060864242 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f41640f5830 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:58.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.179+0000 7f416bbc2700 1 -- 172.21.15.194:0/3060864242 >> 172.21.15.194:0/3060864242 conn(0x7f41640f0b70 msgr2=0x7f41640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:58.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.179+0000 7f416bbc2700 1 -- 172.21.15.194:0/3060864242 shutdown_connections 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.179+0000 7f416bbc2700 1 -- 172.21.15.194:0/3060864242 wait complete. 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.180+0000 7f416bbc2700 1 Processor -- start 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.180+0000 7f416bbc2700 1 -- start start 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.180+0000 7f416bbc2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f4164109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.180+0000 7f416bbc2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4164060240 con 0x7f41640f5410 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.180+0000 7f416995e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f4164109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.180+0000 7f416995e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f4164109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:57364/0 (socket says 172.21.15.194:57364) 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.180+0000 7f416995e700 1 -- 172.21.15.194:0/3231641181 learned_addr learned my addr 172.21.15.194:0/3231641181 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.181+0000 7f416995e700 1 -- 172.21.15.194:0/3231641181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4158005040 con 0x7f41640f5410 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.181+0000 7f416995e700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f4164109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4158013510 tx=0x7f415800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.181+0000 7f41567fc700 1 -- 172.21.15.194:0/3231641181 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4158009320 con 0x7f41640f5410 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.181+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4164060440 con 0x7f41640f5410 2024-01-27T23:26:58.171 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.181+0000 7f41567fc700 1 -- 172.21.15.194:0/3231641181 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4158013f00 con 0x7f41640f5410 2024-01-27T23:26:58.172 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.181+0000 7f41567fc700 1 -- 172.21.15.194:0/3231641181 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4158014d90 con 0x7f41640f5410 2024-01-27T23:26:58.172 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.181+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4164109ae0 con 0x7f41640f5410 2024-01-27T23:26:58.172 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.182+0000 7f41567fc700 1 -- 172.21.15.194:0/3231641181 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4158014420 con 0x7f41640f5410 2024-01-27T23:26:58.172 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.182+0000 7f41567fc700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f415002ff60 0x7f4150032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:26:58.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.182+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4164045ad0 con 0x7f41640f5410 2024-01-27T23:26:58.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.182+0000 7f41567fc700 1 -- 172.21.15.194:0/3231641181 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4158041000 con 0x7f41640f5410 2024-01-27T23:26:58.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.182+0000 7f416915d700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f415002ff60 0x7f4150032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:26:58.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.182+0000 7f416915d700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f415002ff60 0x7f4150032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f41600096a0 tx=0x7f4160006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:26:58.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.186+0000 7f41567fc700 1 -- 172.21.15.194:0/3231641181 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f41580095f0 con 0x7f41640f5410 2024-01-27T23:26:58.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.394+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f41640faf50 con 0x7f41640f5410 2024-01-27T23:26:58.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.395+0000 7f41567fc700 1 -- 172.21.15.194:0/3231641181 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f415801db70 con 0x7f41640f5410 2024-01-27T23:26:58.384 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:26:58.384 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:26:58.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f415002ff60 msgr2=0x7f4150032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f415002ff60 0x7f4150032420 secure :-1 s=READY pgs=35 cs=0 l=1 rev1=1 rx=0x7f41600096a0 tx=0x7f4160006b40).stop 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 msgr2=0x7f4164109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f4164109430 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f4158013510 tx=0x7f415800b040).stop 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 shutdown_connections 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f415002ff60 0x7f4150032420 unknown :-1 s=CLOSED pgs=35 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 --2- 172.21.15.194:0/3231641181 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f41640f5410 0x7f4164109430 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 >> 172.21.15.194:0/3231641181 conn(0x7f41640f0b70 msgr2=0x7f41640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 shutdown_connections 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:26:58.396+0000 7f416bbc2700 1 -- 172.21.15.194:0/3231641181 wait complete. 2024-01-27T23:26:58.386 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:26:58.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:57.271056+0000 mgr.smithi121.hcppvz (mgr.14162) 51 : cluster [DBG] pgmap v44: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:26:59.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:26:59 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:26:58.395958+0000 mon.smithi121 (mon.0) 144 : audit [DBG] from='client.? 172.21.15.194:0/3231641181' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:26:59.802 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:26:59.802 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:00.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:26:59.271410+0000 mgr.smithi121.hcppvz (mgr.14162) 52 : cluster [DBG] pgmap v45: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:00.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:00.999+0000 7fa46b59e700 1 -- 172.21.15.194:0/2929519972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 msgr2=0x7fa46c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:00.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:00.999+0000 7fa46b59e700 1 --2- 172.21.15.194:0/2929519972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c0f55c0 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fa45c009fb0 tx=0x7fa45c009b30).stop 2024-01-27T23:27:00.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 -- 172.21.15.194:0/2929519972 shutdown_connections 2024-01-27T23:27:00.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 --2- 172.21.15.194:0/2929519972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c0f55c0 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 -- 172.21.15.194:0/2929519972 >> 172.21.15.194:0/2929519972 conn(0x7fa46c0f09a0 msgr2=0x7fa46c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 -- 172.21.15.194:0/2929519972 shutdown_connections 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 -- 172.21.15.194:0/2929519972 wait complete. 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 Processor -- start 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 -- start start 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.000+0000 7fa46b59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa46c1069d0 con 0x7fa46c0f51a0 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa46a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa46a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:57374/0 (socket says 172.21.15.194:57374) 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa46a59c700 1 -- 172.21.15.194:0/1592080400 learned_addr learned my addr 172.21.15.194:0/1592080400 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa46a59c700 1 -- 172.21.15.194:0/1592080400 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa45c005040 con 0x7fa46c0f51a0 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa46a59c700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1064e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa45c013510 tx=0x7fa45c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa45affd700 1 -- 172.21.15.194:0/1592080400 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa45c009320 con 0x7fa46c0f51a0 2024-01-27T23:27:00.991 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa45affd700 1 -- 172.21.15.194:0/1592080400 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa45c013f00 con 0x7fa46c0f51a0 2024-01-27T23:27:00.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa46c106bd0 con 0x7fa46c0f51a0 2024-01-27T23:27:00.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa45affd700 1 -- 172.21.15.194:0/1592080400 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa45c014dc0 con 0x7fa46c0f51a0 2024-01-27T23:27:00.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.001+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa46c107060 con 0x7fa46c0f51a0 2024-01-27T23:27:00.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.002+0000 7fa45affd700 1 -- 172.21.15.194:0/1592080400 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa45c0095f0 con 0x7fa46c0f51a0 2024-01-27T23:27:00.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.002+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa46c045ad0 con 0x7fa46c0f51a0 2024-01-27T23:27:00.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.002+0000 7fa45affd700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa45402ffb0 0x7fa454032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:00.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.002+0000 7fa45affd700 1 -- 172.21.15.194:0/1592080400 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa45c040ff0 con 0x7fa46c0f51a0 2024-01-27T23:27:00.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.002+0000 7fa469d9b700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa45402ffb0 0x7fa454032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:00.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.003+0000 7fa469d9b700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa45402ffb0 0x7fa454032470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa4600096a0 tx=0x7fa460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:00.995 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.006+0000 7fa45affd700 1 -- 172.21.15.194:0/1592080400 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa45c06d2b0 con 0x7fa46c0f51a0 2024-01-27T23:27:01.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.212+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa46c107f10 con 0x7fa46c0f51a0 2024-01-27T23:27:01.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.213+0000 7fa45affd700 1 -- 172.21.15.194:0/1592080400 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa45c014420 con 0x7fa46c0f51a0 2024-01-27T23:27:01.202 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:01.202 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.214+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa45402ffb0 msgr2=0x7fa454032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.214+0000 7fa46b59e700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa45402ffb0 0x7fa454032470 secure :-1 s=READY pgs=36 cs=0 l=1 rev1=1 rx=0x7fa4600096a0 tx=0x7fa460006b40).stop 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.214+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 msgr2=0x7fa46c1064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.214+0000 7fa46b59e700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1064e0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fa45c013510 tx=0x7fa45c00b040).stop 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.215+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 shutdown_connections 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.215+0000 7fa46b59e700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa45402ffb0 0x7fa454032470 unknown :-1 s=CLOSED pgs=36 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.215+0000 7fa46b59e700 1 --2- 172.21.15.194:0/1592080400 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa46c0f51a0 0x7fa46c1064e0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.215+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 >> 172.21.15.194:0/1592080400 conn(0x7fa46c0f09a0 msgr2=0x7fa46c05fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.215+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 shutdown_connections 2024-01-27T23:27:01.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:01.215+0000 7fa46b59e700 1 -- 172.21.15.194:0/1592080400 wait complete. 2024-01-27T23:27:01.205 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:01.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:01 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:01.214318+0000 mon.smithi121 (mon.0) 145 : audit [DBG] from='client.? 172.21.15.194:0/1592080400' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:02.583 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:02.583 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:02.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:01.271777+0000 mgr.smithi121.hcppvz (mgr.14162) 53 : cluster [DBG] pgmap v46: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:03.759 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.770+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1521724074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 msgr2=0x7f1cc80f55e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:03.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.770+0000 7f1ccd7ba700 1 --2- 172.21.15.194:0/1521724074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80f55e0 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f1cb8009fb0 tx=0x7f1cb8009b30).stop 2024-01-27T23:27:03.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.770+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1521724074 shutdown_connections 2024-01-27T23:27:03.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.770+0000 7f1ccd7ba700 1 --2- 172.21.15.194:0/1521724074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80f55e0 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:03.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.770+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1521724074 >> 172.21.15.194:0/1521724074 conn(0x7f1cc80f0920 msgr2=0x7f1cc80f2d40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:03.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.770+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1521724074 shutdown_connections 2024-01-27T23:27:03.760 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.770+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1521724074 wait complete. 2024-01-27T23:27:03.762 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.771+0000 7f1ccd7ba700 1 Processor -- start 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.771+0000 7f1ccd7ba700 1 -- start start 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.771+0000 7f1ccd7ba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80fd1c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.771+0000 7f1ccd7ba700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cc80fb640 con 0x7f1cc80f51c0 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.771+0000 7f1cc6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80fd1c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.771+0000 7f1cc6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80fd1c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:57402/0 (socket says 172.21.15.194:57402) 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.771+0000 7f1cc6ffd700 1 -- 172.21.15.194:0/1011686559 learned_addr learned my addr 172.21.15.194:0/1011686559 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.772+0000 7f1cc6ffd700 1 -- 172.21.15.194:0/1011686559 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1cb8005040 con 0x7f1cc80f51c0 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.772+0000 7f1cc6ffd700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80fd1c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1cb8013510 tx=0x7f1cb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.772+0000 7f1caf7fe700 1 -- 172.21.15.194:0/1011686559 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb8009320 con 0x7f1cc80f51c0 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.772+0000 7f1caf7fe700 1 -- 172.21.15.194:0/1011686559 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1cb8013f00 con 0x7f1cc80f51c0 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.772+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cc80fb7e0 con 0x7f1cc80f51c0 2024-01-27T23:27:03.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.772+0000 7f1caf7fe700 1 -- 172.21.15.194:0/1011686559 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1cb8014d90 con 0x7f1cc80f51c0 2024-01-27T23:27:03.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.772+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cc80fbc70 con 0x7f1cc80f51c0 2024-01-27T23:27:03.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.773+0000 7f1caf7fe700 1 -- 172.21.15.194:0/1011686559 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1cb8014420 con 0x7f1cc80f51c0 2024-01-27T23:27:03.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.773+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cc8059100 con 0x7f1cc80f51c0 2024-01-27T23:27:03.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.773+0000 7f1caf7fe700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1cb0030000 0x7f1cb00324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:03.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.773+0000 7f1caf7fe700 1 -- 172.21.15.194:0/1011686559 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1cb80410b0 con 0x7f1cc80f51c0 2024-01-27T23:27:03.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.773+0000 7f1cc67fc700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1cb0030000 0x7f1cb00324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:03.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.774+0000 7f1cc67fc700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1cb0030000 0x7f1cb00324c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1cbc0096a0 tx=0x7f1cbc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:03.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.778+0000 7f1caf7fe700 1 -- 172.21.15.194:0/1011686559 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1cb80095f0 con 0x7f1cc80f51c0 2024-01-27T23:27:03.974 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.985+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cc8045ad0 con 0x7f1cc80f51c0 2024-01-27T23:27:03.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.986+0000 7f1caf7fe700 1 -- 172.21.15.194:0/1011686559 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1cb801db70 con 0x7f1cc80f51c0 2024-01-27T23:27:03.975 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:03.975 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1cb0030000 msgr2=0x7f1cb00324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1cb0030000 0x7f1cb00324c0 secure :-1 s=READY pgs=37 cs=0 l=1 rev1=1 rx=0x7f1cbc0096a0 tx=0x7f1cbc006b40).stop 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 msgr2=0x7f1cc80fd1c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80fd1c0 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f1cb8013510 tx=0x7f1cb800b040).stop 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 shutdown_connections 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1cb0030000 0x7f1cb00324c0 unknown :-1 s=CLOSED pgs=37 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 --2- 172.21.15.194:0/1011686559 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cc80f51c0 0x7f1cc80fd1c0 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 >> 172.21.15.194:0/1011686559 conn(0x7f1cc80f0920 msgr2=0x7f1cc805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 shutdown_connections 2024-01-27T23:27:03.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:03.987+0000 7f1ccd7ba700 1 -- 172.21.15.194:0/1011686559 wait complete. 2024-01-27T23:27:03.978 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:04.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:03.272066+0000 mgr.smithi121.hcppvz (mgr.14162) 54 : cluster [DBG] pgmap v47: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:04.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:03.986977+0000 mon.smithi121 (mon.0) 146 : audit [DBG] from='client.? 172.21.15.194:0/1011686559' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:05.341 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:05.342 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:06.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.570+0000 7f592f2c6700 1 -- 172.21.15.194:0/3480486993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 msgr2=0x7f59280f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:06.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.570+0000 7f592f2c6700 1 --2- 172.21.15.194:0/3480486993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f59280f4be0 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f5918009fb0 tx=0x7f5918009b30).stop 2024-01-27T23:27:06.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.570+0000 7f592f2c6700 1 -- 172.21.15.194:0/3480486993 shutdown_connections 2024-01-27T23:27:06.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.570+0000 7f592f2c6700 1 --2- 172.21.15.194:0/3480486993 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f59280f4be0 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.570+0000 7f592f2c6700 1 -- 172.21.15.194:0/3480486993 >> 172.21.15.194:0/3480486993 conn(0x7f59280f02e0 msgr2=0x7f59280f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.570+0000 7f592f2c6700 1 -- 172.21.15.194:0/3480486993 shutdown_connections 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.570+0000 7f592f2c6700 1 -- 172.21.15.194:0/3480486993 wait complete. 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592f2c6700 1 Processor -- start 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592f2c6700 1 -- start start 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592f2c6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f5928106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592f2c6700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5928106bf0 con 0x7f59280f47c0 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592d062700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f5928106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592d062700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f5928106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:46300/0 (socket says 172.21.15.194:46300) 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592d062700 1 -- 172.21.15.194:0/922835611 learned_addr learned my addr 172.21.15.194:0/922835611 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.571+0000 7f592d062700 1 -- 172.21.15.194:0/922835611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5918005040 con 0x7f59280f47c0 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.572+0000 7f592d062700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f5928106700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5918013510 tx=0x7f591800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:06.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.572+0000 7f591dffb700 1 -- 172.21.15.194:0/922835611 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5918009320 con 0x7f59280f47c0 2024-01-27T23:27:06.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.572+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5928106df0 con 0x7f59280f47c0 2024-01-27T23:27:06.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.572+0000 7f591dffb700 1 -- 172.21.15.194:0/922835611 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5918013f00 con 0x7f59280f47c0 2024-01-27T23:27:06.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.572+0000 7f591dffb700 1 -- 172.21.15.194:0/922835611 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5918014d90 con 0x7f59280f47c0 2024-01-27T23:27:06.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.572+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5928107280 con 0x7f59280f47c0 2024-01-27T23:27:06.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.573+0000 7f591dffb700 1 -- 172.21.15.194:0/922835611 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5918014420 con 0x7f59280f47c0 2024-01-27T23:27:06.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.573+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5928045ad0 con 0x7f59280f47c0 2024-01-27T23:27:06.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.573+0000 7f591dffb700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f591402ff60 0x7f5914032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:06.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.573+0000 7f591dffb700 1 -- 172.21.15.194:0/922835611 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5918041050 con 0x7f59280f47c0 2024-01-27T23:27:06.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.573+0000 7f592c861700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f591402ff60 0x7f5914032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:06.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.573+0000 7f592c861700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f591402ff60 0x7f5914032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:06.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.577+0000 7f591dffb700 1 -- 172.21.15.194:0/922835611 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59180095f0 con 0x7f59280f47c0 2024-01-27T23:27:06.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:05.272309+0000 mgr.smithi121.hcppvz (mgr.14162) 55 : cluster [DBG] pgmap v48: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:06.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.784+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59281080d0 con 0x7f59280f47c0 2024-01-27T23:27:06.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.785+0000 7f591dffb700 1 -- 172.21.15.194:0/922835611 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f591801db70 con 0x7f59280f47c0 2024-01-27T23:27:06.774 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:06.774 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f591402ff60 msgr2=0x7f5914032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f591402ff60 0x7f5914032420 secure :-1 s=READY pgs=38 cs=0 l=1 rev1=1 rx=0x7f59240096a0 tx=0x7f5924006b40).stop 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 msgr2=0x7f5928106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f5928106700 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7f5918013510 tx=0x7f591800b040).stop 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 shutdown_connections 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f591402ff60 0x7f5914032420 unknown :-1 s=CLOSED pgs=38 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 --2- 172.21.15.194:0/922835611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59280f47c0 0x7f5928106700 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.786+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 >> 172.21.15.194:0/922835611 conn(0x7f59280f02e0 msgr2=0x7f59280f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.787+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 shutdown_connections 2024-01-27T23:27:06.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:06.787+0000 7f592f2c6700 1 -- 172.21.15.194:0/922835611 wait complete. 2024-01-27T23:27:06.777 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:07.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:07 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:06.786239+0000 mon.smithi121 (mon.0) 147 : audit [DBG] from='client.? 172.21.15.194:0/922835611' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:08.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:08.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:08.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:07.272509+0000 mgr.smithi121.hcppvz (mgr.14162) 56 : cluster [DBG] pgmap v49: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:09.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.339+0000 7fce09820700 1 -- 172.21.15.194:0/3689129110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 msgr2=0x7fce040f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:09.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.339+0000 7fce09820700 1 --2- 172.21.15.194:0/3689129110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f4be0 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7fcdf8009fb0 tx=0x7fcdf8009b30).stop 2024-01-27T23:27:09.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.339+0000 7fce09820700 1 -- 172.21.15.194:0/3689129110 shutdown_connections 2024-01-27T23:27:09.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.339+0000 7fce09820700 1 --2- 172.21.15.194:0/3689129110 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f4be0 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:09.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.339+0000 7fce09820700 1 -- 172.21.15.194:0/3689129110 >> 172.21.15.194:0/3689129110 conn(0x7fce040f02e0 msgr2=0x7fce040f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:09.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.339+0000 7fce09820700 1 -- 172.21.15.194:0/3689129110 shutdown_connections 2024-01-27T23:27:09.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.339+0000 7fce09820700 1 -- 172.21.15.194:0/3689129110 wait complete. 2024-01-27T23:27:09.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.340+0000 7fce09820700 1 Processor -- start 2024-01-27T23:27:09.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.340+0000 7fce09820700 1 -- start start 2024-01-27T23:27:09.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.340+0000 7fce09820700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f7370 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:09.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.340+0000 7fce09820700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fce040f57f0 con 0x7fce040f47c0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.340+0000 7fce02ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f7370 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.340+0000 7fce02ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f7370 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:46316/0 (socket says 172.21.15.194:46316) 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.340+0000 7fce02ffd700 1 -- 172.21.15.194:0/3092311020 learned_addr learned my addr 172.21.15.194:0/3092311020 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.341+0000 7fce02ffd700 1 -- 172.21.15.194:0/3092311020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcdf8005040 con 0x7fce040f47c0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.341+0000 7fce02ffd700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f7370 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcdf8013510 tx=0x7fcdf800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.341+0000 7fcdeb7fe700 1 -- 172.21.15.194:0/3092311020 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdf8009320 con 0x7fce040f47c0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.341+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fce040f5990 con 0x7fce040f47c0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.341+0000 7fcdeb7fe700 1 -- 172.21.15.194:0/3092311020 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcdf8013f00 con 0x7fce040f47c0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.341+0000 7fcdeb7fe700 1 -- 172.21.15.194:0/3092311020 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcdf8014d90 con 0x7fce040f47c0 2024-01-27T23:27:09.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.341+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fce040f5e20 con 0x7fce040f47c0 2024-01-27T23:27:09.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.342+0000 7fcdeb7fe700 1 -- 172.21.15.194:0/3092311020 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcdf80263a0 con 0x7fce040f47c0 2024-01-27T23:27:09.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.342+0000 7fcdeb7fe700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcdec02ff60 0x7fcdec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:09.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.342+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fce04059100 con 0x7fce040f47c0 2024-01-27T23:27:09.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.342+0000 7fcdeb7fe700 1 -- 172.21.15.194:0/3092311020 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcdf8041300 con 0x7fce040f47c0 2024-01-27T23:27:09.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.342+0000 7fce027fc700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcdec02ff60 0x7fcdec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:09.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.342+0000 7fce027fc700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcdec02ff60 0x7fcdec032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcdf40096a0 tx=0x7fcdf4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:09.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.346+0000 7fcdeb7fe700 1 -- 172.21.15.194:0/3092311020 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcdf80126f0 con 0x7fce040f47c0 2024-01-27T23:27:09.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.551+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fce040fbf60 con 0x7fce040f47c0 2024-01-27T23:27:09.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.552+0000 7fcdeb7fe700 1 -- 172.21.15.194:0/3092311020 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcdf8012330 con 0x7fce040f47c0 2024-01-27T23:27:09.541 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:09.541 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:09.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcdec02ff60 msgr2=0x7fcdec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:09.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcdec02ff60 0x7fcdec032420 secure :-1 s=READY pgs=39 cs=0 l=1 rev1=1 rx=0x7fcdf40096a0 tx=0x7fcdf4006b40).stop 2024-01-27T23:27:09.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 msgr2=0x7fce040f7370 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:09.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f7370 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7fcdf8013510 tx=0x7fcdf800b040).stop 2024-01-27T23:27:09.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 shutdown_connections 2024-01-27T23:27:09.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcdec02ff60 0x7fcdec032420 unknown :-1 s=CLOSED pgs=39 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:09.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 --2- 172.21.15.194:0/3092311020 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fce040f47c0 0x7fce040f7370 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:09.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 >> 172.21.15.194:0/3092311020 conn(0x7fce040f02e0 msgr2=0x7fce040f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:09.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 shutdown_connections 2024-01-27T23:27:09.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:09.554+0000 7fce09820700 1 -- 172.21.15.194:0/3092311020 wait complete. 2024-01-27T23:27:09.544 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:10.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:09.272862+0000 mgr.smithi121.hcppvz (mgr.14162) 57 : cluster [DBG] pgmap v50: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:10.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:09.553272+0000 mon.smithi121 (mon.0) 148 : audit [DBG] from='client.? 172.21.15.194:0/3092311020' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:10.956 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:10.956 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:12.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.067+0000 7fda3191e700 1 -- 172.21.15.194:0/931306652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 msgr2=0x7fda2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:12.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.067+0000 7fda3191e700 1 --2- 172.21.15.194:0/931306652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c0f5830 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fda1c009fb0 tx=0x7fda1c009b30).stop 2024-01-27T23:27:12.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.067+0000 7fda3191e700 1 -- 172.21.15.194:0/931306652 shutdown_connections 2024-01-27T23:27:12.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.067+0000 7fda3191e700 1 --2- 172.21.15.194:0/931306652 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c0f5830 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:12.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.067+0000 7fda3191e700 1 -- 172.21.15.194:0/931306652 >> 172.21.15.194:0/931306652 conn(0x7fda2c0f0b70 msgr2=0x7fda2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:12.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.067+0000 7fda3191e700 1 -- 172.21.15.194:0/931306652 shutdown_connections 2024-01-27T23:27:12.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.067+0000 7fda3191e700 1 -- 172.21.15.194:0/931306652 wait complete. 2024-01-27T23:27:12.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.068+0000 7fda3191e700 1 Processor -- start 2024-01-27T23:27:12.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.068+0000 7fda3191e700 1 -- start start 2024-01-27T23:27:12.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.068+0000 7fda3191e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:12.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.068+0000 7fda3191e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fda2c106bf0 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda2affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda2affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:46334/0 (socket says 172.21.15.194:46334) 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda2affd700 1 -- 172.21.15.194:0/3515106951 learned_addr learned my addr 172.21.15.194:0/3515106951 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda2affd700 1 -- 172.21.15.194:0/3515106951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fda1c005040 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda2affd700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c106700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fda1c013510 tx=0x7fda1c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda137fe700 1 -- 172.21.15.194:0/3515106951 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1c009320 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fda2c106df0 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda137fe700 1 -- 172.21.15.194:0/3515106951 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fda1c013f00 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda137fe700 1 -- 172.21.15.194:0/3515106951 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fda1c014d90 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.069+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fda2c107280 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.070+0000 7fda137fe700 1 -- 172.21.15.194:0/3515106951 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fda1c0095f0 con 0x7fda2c0f5410 2024-01-27T23:27:12.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.070+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fda2c045ad0 con 0x7fda2c0f5410 2024-01-27T23:27:12.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.070+0000 7fda137fe700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fda1402ff60 0x7fda14032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:12.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.070+0000 7fda137fe700 1 -- 172.21.15.194:0/3515106951 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fda1c040fe0 con 0x7fda2c0f5410 2024-01-27T23:27:12.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.070+0000 7fda2a7fc700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fda1402ff60 0x7fda14032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:12.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.071+0000 7fda2a7fc700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fda1402ff60 0x7fda14032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fda200096a0 tx=0x7fda20006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:12.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.075+0000 7fda137fe700 1 -- 172.21.15.194:0/3515106951 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fda1c01d7f0 con 0x7fda2c0f5410 2024-01-27T23:27:12.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.282+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fda2c0fae60 con 0x7fda2c0f5410 2024-01-27T23:27:12.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.282+0000 7fda137fe700 1 -- 172.21.15.194:0/3515106951 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fda1c010230 con 0x7fda2c0f5410 2024-01-27T23:27:12.272 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:12.272 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fda1402ff60 msgr2=0x7fda14032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fda1402ff60 0x7fda14032420 secure :-1 s=READY pgs=40 cs=0 l=1 rev1=1 rx=0x7fda200096a0 tx=0x7fda20006b40).stop 2024-01-27T23:27:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 msgr2=0x7fda2c106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c106700 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fda1c013510 tx=0x7fda1c00b040).stop 2024-01-27T23:27:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 shutdown_connections 2024-01-27T23:27:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fda1402ff60 0x7fda14032420 unknown :-1 s=CLOSED pgs=40 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:12.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 --2- 172.21.15.194:0/3515106951 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fda2c0f5410 0x7fda2c106700 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:12.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 >> 172.21.15.194:0/3515106951 conn(0x7fda2c0f0b70 msgr2=0x7fda2c0f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:12.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 shutdown_connections 2024-01-27T23:27:12.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:12.284+0000 7fda3191e700 1 -- 172.21.15.194:0/3515106951 wait complete. 2024-01-27T23:27:12.274 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:12.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:11.273118+0000 mgr.smithi121.hcppvz (mgr.14162) 58 : cluster [DBG] pgmap v51: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:13.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:13 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:12.283705+0000 mon.smithi121 (mon.0) 149 : audit [DBG] from='client.? 172.21.15.194:0/3515106951' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:13.647 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:13.648 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:14.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:13.273370+0000 mgr.smithi121.hcppvz (mgr.14162) 59 : cluster [DBG] pgmap v52: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:15.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 -- 172.21.15.194:0/1621123687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 msgr2=0x7f7dd40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:15.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 --2- 172.21.15.194:0/1621123687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd40f66c0 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7f7dc4009fb0 tx=0x7f7dc4009b30).stop 2024-01-27T23:27:15.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 -- 172.21.15.194:0/1621123687 shutdown_connections 2024-01-27T23:27:15.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 --2- 172.21.15.194:0/1621123687 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd40f66c0 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:15.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 -- 172.21.15.194:0/1621123687 >> 172.21.15.194:0/1621123687 conn(0x7f7dd40f1aa0 msgr2=0x7f7dd40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:15.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 -- 172.21.15.194:0/1621123687 shutdown_connections 2024-01-27T23:27:15.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 -- 172.21.15.194:0/1621123687 wait complete. 2024-01-27T23:27:15.108 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 Processor -- start 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 -- start start 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.117+0000 7f7dd884a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7dd41069a0 con 0x7f7dd40f62a0 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dd2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dd2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:46358/0 (socket says 172.21.15.194:46358) 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dd2d9d700 1 -- 172.21.15.194:0/596102768 learned_addr learned my addr 172.21.15.194:0/596102768 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dd2d9d700 1 -- 172.21.15.194:0/596102768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7dc4005040 con 0x7f7dd40f62a0 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dd2d9d700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd41064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7dc4013510 tx=0x7f7dc400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dbb7fe700 1 -- 172.21.15.194:0/596102768 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc4009320 con 0x7f7dd40f62a0 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7dd4106ba0 con 0x7f7dd40f62a0 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dbb7fe700 1 -- 172.21.15.194:0/596102768 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7dc4013f00 con 0x7f7dd40f62a0 2024-01-27T23:27:15.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dbb7fe700 1 -- 172.21.15.194:0/596102768 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7dc4014dc0 con 0x7f7dd40f62a0 2024-01-27T23:27:15.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.118+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7dd4107030 con 0x7f7dd40f62a0 2024-01-27T23:27:15.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.119+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7dd4045ad0 con 0x7f7dd40f62a0 2024-01-27T23:27:15.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.119+0000 7f7dbb7fe700 1 -- 172.21.15.194:0/596102768 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7dc4014420 con 0x7f7dd40f62a0 2024-01-27T23:27:15.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.119+0000 7f7dbb7fe700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7dbc02ff60 0x7f7dbc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:15.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.119+0000 7f7dbb7fe700 1 -- 172.21.15.194:0/596102768 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7dc4041110 con 0x7f7dd40f62a0 2024-01-27T23:27:15.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.119+0000 7f7dd259c700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7dbc02ff60 0x7f7dbc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:15.110 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.120+0000 7f7dd259c700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7dbc02ff60 0x7f7dbc032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7dc80096a0 tx=0x7f7dc8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:15.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.124+0000 7f7dbb7fe700 1 -- 172.21.15.194:0/596102768 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7dc40095f0 con 0x7f7dd40f62a0 2024-01-27T23:27:15.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.331+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7dd4107ee0 con 0x7f7dd40f62a0 2024-01-27T23:27:15.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.331+0000 7f7dbb7fe700 1 -- 172.21.15.194:0/596102768 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7dc401cb10 con 0x7f7dd40f62a0 2024-01-27T23:27:15.321 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:15.321 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:15.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7dbc02ff60 msgr2=0x7f7dbc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:15.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7dbc02ff60 0x7f7dbc032420 secure :-1 s=READY pgs=41 cs=0 l=1 rev1=1 rx=0x7f7dc80096a0 tx=0x7f7dc8006b40).stop 2024-01-27T23:27:15.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 msgr2=0x7f7dd41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:15.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd41064b0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f7dc4013510 tx=0x7f7dc400b040).stop 2024-01-27T23:27:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 shutdown_connections 2024-01-27T23:27:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7dbc02ff60 0x7f7dbc032420 unknown :-1 s=CLOSED pgs=41 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 --2- 172.21.15.194:0/596102768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7dd40f62a0 0x7f7dd41064b0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 >> 172.21.15.194:0/596102768 conn(0x7f7dd40f1aa0 msgr2=0x7f7dd40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 shutdown_connections 2024-01-27T23:27:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:15.333+0000 7f7dd884a700 1 -- 172.21.15.194:0/596102768 wait complete. 2024-01-27T23:27:15.323 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:16.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:15.273717+0000 mgr.smithi121.hcppvz (mgr.14162) 60 : cluster [DBG] pgmap v53: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:16.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:15.332728+0000 mon.smithi121 (mon.0) 150 : audit [DBG] from='client.? 172.21.15.194:0/596102768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:16.720 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:16.720 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:17.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.933+0000 7f2544ed8700 1 -- 172.21.15.194:0/1381428971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 msgr2=0x7f25400f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:17.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.933+0000 7f2544ed8700 1 --2- 172.21.15.194:0/1381428971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f25400f5830 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f2530009fb0 tx=0x7f2530009b30).stop 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.933+0000 7f2544ed8700 1 -- 172.21.15.194:0/1381428971 shutdown_connections 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.933+0000 7f2544ed8700 1 --2- 172.21.15.194:0/1381428971 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f25400f5830 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.933+0000 7f2544ed8700 1 -- 172.21.15.194:0/1381428971 >> 172.21.15.194:0/1381428971 conn(0x7f25400f0b70 msgr2=0x7f25400f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.933+0000 7f2544ed8700 1 -- 172.21.15.194:0/1381428971 shutdown_connections 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.933+0000 7f2544ed8700 1 -- 172.21.15.194:0/1381428971 wait complete. 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.934+0000 7f2544ed8700 1 Processor -- start 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.934+0000 7f2544ed8700 1 -- start start 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.934+0000 7f2544ed8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f2540109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.934+0000 7f2544ed8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2540060220 con 0x7f25400f5410 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f253e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f2540109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:17.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f253e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f2540109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37424/0 (socket says 172.21.15.194:37424) 2024-01-27T23:27:17.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f253e59c700 1 -- 172.21.15.194:0/977517918 learned_addr learned my addr 172.21.15.194:0/977517918 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:17.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f253e59c700 1 -- 172.21.15.194:0/977517918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2530005040 con 0x7f25400f5410 2024-01-27T23:27:17.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f253e59c700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f2540109420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2530013510 tx=0x7f253000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:17.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f2536ffd700 1 -- 172.21.15.194:0/977517918 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2530009320 con 0x7f25400f5410 2024-01-27T23:27:17.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f2536ffd700 1 -- 172.21.15.194:0/977517918 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2530013f00 con 0x7f25400f5410 2024-01-27T23:27:17.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2540060420 con 0x7f25400f5410 2024-01-27T23:27:17.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f2536ffd700 1 -- 172.21.15.194:0/977517918 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2530014d90 con 0x7f25400f5410 2024-01-27T23:27:17.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.935+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2540109ad0 con 0x7f25400f5410 2024-01-27T23:27:17.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.936+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2540059100 con 0x7f25400f5410 2024-01-27T23:27:17.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.936+0000 7f2536ffd700 1 -- 172.21.15.194:0/977517918 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25300095f0 con 0x7f25400f5410 2024-01-27T23:27:17.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.936+0000 7f2536ffd700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f252c02ffb0 0x7f252c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:17.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.937+0000 7f2536ffd700 1 -- 172.21.15.194:0/977517918 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f253001b920 con 0x7f25400f5410 2024-01-27T23:27:17.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.937+0000 7f253dd9b700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f252c02ffb0 0x7f252c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:17.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.937+0000 7f253dd9b700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f252c02ffb0 0x7f252c032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f25280096a0 tx=0x7f2528006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:17.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:17.941+0000 7f2536ffd700 1 -- 172.21.15.194:0/977517918 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2530014420 con 0x7f25400f5410 2024-01-27T23:27:18.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.147+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f25400fae60 con 0x7f25400f5410 2024-01-27T23:27:18.136 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.147+0000 7f2536ffd700 1 -- 172.21.15.194:0/977517918 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f253003fe60 con 0x7f25400f5410 2024-01-27T23:27:18.137 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:18.137 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:18.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f252c02ffb0 msgr2=0x7f252c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:18.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f252c02ffb0 0x7f252c032470 secure :-1 s=READY pgs=42 cs=0 l=1 rev1=1 rx=0x7f25280096a0 tx=0x7f2528006b40).stop 2024-01-27T23:27:18.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 msgr2=0x7f2540109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:18.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f2540109420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f2530013510 tx=0x7f253000b040).stop 2024-01-27T23:27:18.138 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 shutdown_connections 2024-01-27T23:27:18.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f252c02ffb0 0x7f252c032470 unknown :-1 s=CLOSED pgs=42 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:18.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 --2- 172.21.15.194:0/977517918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25400f5410 0x7f2540109420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:18.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 >> 172.21.15.194:0/977517918 conn(0x7f25400f0b70 msgr2=0x7f25400f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:18.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 shutdown_connections 2024-01-27T23:27:18.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:18.149+0000 7f2544ed8700 1 -- 172.21.15.194:0/977517918 wait complete. 2024-01-27T23:27:18.139 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:18.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:17.273977+0000 mgr.smithi121.hcppvz (mgr.14162) 61 : cluster [DBG] pgmap v54: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:18.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:17.393050+0000 mon.smithi121 (mon.0) 151 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:27:18.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:17.401745+0000 mon.smithi121 (mon.0) 152 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:27:18.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:18.148742+0000 mon.smithi121 (mon.0) 153 : audit [DBG] from='client.? 172.21.15.194:0/977517918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:19.521 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:19.521 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:20.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:19.274321+0000 mgr.smithi121.hcppvz (mgr.14162) 62 : cluster [DBG] pgmap v55: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:20.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.734+0000 7f1005806700 1 -- 172.21.15.194:0/2231501002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 msgr2=0x7f10000f6eb0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:20.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.734+0000 7f1005806700 1 --2- 172.21.15.194:0/2231501002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10000f6eb0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f0ff4009fb0 tx=0x7f0ff4009b30).stop 2024-01-27T23:27:20.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.734+0000 7f1005806700 1 -- 172.21.15.194:0/2231501002 shutdown_connections 2024-01-27T23:27:20.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.734+0000 7f1005806700 1 --2- 172.21.15.194:0/2231501002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10000f6eb0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:20.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.734+0000 7f1005806700 1 -- 172.21.15.194:0/2231501002 >> 172.21.15.194:0/2231501002 conn(0x7f10000f02e0 msgr2=0x7f10000f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:20.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.734+0000 7f1005806700 1 -- 172.21.15.194:0/2231501002 shutdown_connections 2024-01-27T23:27:20.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.734+0000 7f1005806700 1 -- 172.21.15.194:0/2231501002 wait complete. 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.735+0000 7f1005806700 1 Processor -- start 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.735+0000 7f1005806700 1 -- start start 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.735+0000 7f1005806700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10001088a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.735+0000 7f1005806700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1000108d90 con 0x7f10000f6a90 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.735+0000 7f0ffeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10001088a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f0ffeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10001088a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37442/0 (socket says 172.21.15.194:37442) 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f0ffeffd700 1 -- 172.21.15.194:0/704231885 learned_addr learned my addr 172.21.15.194:0/704231885 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:20.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f0ffeffd700 1 -- 172.21.15.194:0/704231885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0ff4005040 con 0x7f10000f6a90 2024-01-27T23:27:20.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f0ffeffd700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10001088a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0ff4013510 tx=0x7f0ff400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:20.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f0fe77fe700 1 -- 172.21.15.194:0/704231885 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ff4009320 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f0fe77fe700 1 -- 172.21.15.194:0/704231885 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0ff4013f00 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1000108f90 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f0fe77fe700 1 -- 172.21.15.194:0/704231885 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0ff4014dc0 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.736+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1000109420 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.737+0000 7f0fe77fe700 1 -- 172.21.15.194:0/704231885 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0ff401d3d0 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.737+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1000045ad0 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.737+0000 7f0fe77fe700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0fe802ff60 0x7f0fe8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.737+0000 7f0fe77fe700 1 -- 172.21.15.194:0/704231885 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0ff4041010 con 0x7f10000f6a90 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.737+0000 7f0ffe7fc700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0fe802ff60 0x7f0fe8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:20.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.738+0000 7f0ffe7fc700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0fe802ff60 0x7f0fe8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0ff00096a0 tx=0x7f0ff0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:20.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.742+0000 7f0fe77fe700 1 -- 172.21.15.194:0/704231885 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0ff40126f0 con 0x7f10000f6a90 2024-01-27T23:27:20.935 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.946+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f100010a2d0 con 0x7f10000f6a90 2024-01-27T23:27:20.936 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.947+0000 7f0fe77fe700 1 -- 172.21.15.194:0/704231885 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0ff4012330 con 0x7f10000f6a90 2024-01-27T23:27:20.936 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:20.939 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:20.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0fe802ff60 msgr2=0x7f0fe8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:20.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0fe802ff60 0x7f0fe8032420 secure :-1 s=READY pgs=43 cs=0 l=1 rev1=1 rx=0x7f0ff00096a0 tx=0x7f0ff0006b40).stop 2024-01-27T23:27:20.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 msgr2=0x7f10001088a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10001088a0 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f0ff4013510 tx=0x7f0ff400b040).stop 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 shutdown_connections 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0fe802ff60 0x7f0fe8032420 unknown :-1 s=CLOSED pgs=43 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 --2- 172.21.15.194:0/704231885 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f10000f6a90 0x7f10001088a0 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 >> 172.21.15.194:0/704231885 conn(0x7f10000f02e0 msgr2=0x7f10000f0dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 shutdown_connections 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:20.948+0000 7f1005806700 1 -- 172.21.15.194:0/704231885 wait complete. 2024-01-27T23:27:20.940 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:21.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:21 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:20.948054+0000 mon.smithi121 (mon.0) 154 : audit [DBG] from='client.? 172.21.15.194:0/704231885' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:22.351 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:22.351 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:22.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:21.274679+0000 mgr.smithi121.hcppvz (mgr.14162) 63 : cluster [DBG] pgmap v56: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.511+0000 7f577eb71700 1 -- 172.21.15.194:0/1614096148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 msgr2=0x7f57780f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.511+0000 7f577eb71700 1 --2- 172.21.15.194:0/1614096148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f57780f5830 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f576c009fb0 tx=0x7f576c009b30).stop 2024-01-27T23:27:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.511+0000 7f577eb71700 1 -- 172.21.15.194:0/1614096148 shutdown_connections 2024-01-27T23:27:23.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.511+0000 7f577eb71700 1 --2- 172.21.15.194:0/1614096148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f57780f5830 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.511+0000 7f577eb71700 1 -- 172.21.15.194:0/1614096148 >> 172.21.15.194:0/1614096148 conn(0x7f57780f0b70 msgr2=0x7f57780f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.511+0000 7f577eb71700 1 -- 172.21.15.194:0/1614096148 shutdown_connections 2024-01-27T23:27:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.511+0000 7f577eb71700 1 -- 172.21.15.194:0/1614096148 wait complete. 2024-01-27T23:27:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.512+0000 7f577eb71700 1 Processor -- start 2024-01-27T23:27:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.512+0000 7f577eb71700 1 -- start start 2024-01-27T23:27:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.512+0000 7f577eb71700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f5778105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:23.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.512+0000 7f577eb71700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5778105b20 con 0x7f57780f5410 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.512+0000 7f577c90d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f5778105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.512+0000 7f577c90d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f5778105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37452/0 (socket says 172.21.15.194:37452) 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.512+0000 7f577c90d700 1 -- 172.21.15.194:0/922709523 learned_addr learned my addr 172.21.15.194:0/922709523 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.513+0000 7f577c90d700 1 -- 172.21.15.194:0/922709523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f576c005040 con 0x7f57780f5410 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.513+0000 7f577c90d700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f5778105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f576c013510 tx=0x7f576c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.513+0000 7f57757fa700 1 -- 172.21.15.194:0/922709523 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f576c009320 con 0x7f57780f5410 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.513+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5778105d20 con 0x7f57780f5410 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.513+0000 7f57757fa700 1 -- 172.21.15.194:0/922709523 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f576c013f00 con 0x7f57780f5410 2024-01-27T23:27:23.504 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.513+0000 7f57757fa700 1 -- 172.21.15.194:0/922709523 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f576c014d90 con 0x7f57780f5410 2024-01-27T23:27:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.513+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57781061b0 con 0x7f57780f5410 2024-01-27T23:27:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.514+0000 7f57757fa700 1 -- 172.21.15.194:0/922709523 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f576c0263a0 con 0x7f57780f5410 2024-01-27T23:27:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.514+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5778045ad0 con 0x7f57780f5410 2024-01-27T23:27:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.514+0000 7f57757fa700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f576802ffb0 0x7f5768032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.514+0000 7f57757fa700 1 -- 172.21.15.194:0/922709523 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f576c040fd0 con 0x7f57780f5410 2024-01-27T23:27:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.514+0000 7f5777fff700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f576802ffb0 0x7f5768032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:23.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.514+0000 7f5777fff700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f576802ffb0 0x7f5768032470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f57640096a0 tx=0x7f5764006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:23.508 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.519+0000 7f57757fa700 1 -- 172.21.15.194:0/922709523 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f576c06d2b0 con 0x7f57780f5410 2024-01-27T23:27:23.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.724+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57780fae60 con 0x7f57780f5410 2024-01-27T23:27:23.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.725+0000 7f57757fa700 1 -- 172.21.15.194:0/922709523 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f576c0095f0 con 0x7f57780f5410 2024-01-27T23:27:23.715 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:23.715 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:23.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f576802ffb0 msgr2=0x7f5768032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:23.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f576802ffb0 0x7f5768032470 secure :-1 s=READY pgs=44 cs=0 l=1 rev1=1 rx=0x7f57640096a0 tx=0x7f5764006b40).stop 2024-01-27T23:27:23.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 msgr2=0x7f5778105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:23.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f5778105630 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f576c013510 tx=0x7f576c00b040).stop 2024-01-27T23:27:23.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 shutdown_connections 2024-01-27T23:27:23.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f576802ffb0 0x7f5768032470 unknown :-1 s=CLOSED pgs=44 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:23.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 --2- 172.21.15.194:0/922709523 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57780f5410 0x7f5778105630 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:23.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 >> 172.21.15.194:0/922709523 conn(0x7f57780f0b70 msgr2=0x7f57780f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:23.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 shutdown_connections 2024-01-27T23:27:23.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:23.727+0000 7f577eb71700 1 -- 172.21.15.194:0/922709523 wait complete. 2024-01-27T23:27:23.717 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:24.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:23.275025+0000 mgr.smithi121.hcppvz (mgr.14162) 64 : cluster [DBG] pgmap v57: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:24.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:23.726431+0000 mon.smithi121 (mon.0) 155 : audit [DBG] from='client.? 172.21.15.194:0/922709523' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:25.138 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:25.138 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:26.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.324+0000 7f2439216700 1 -- 172.21.15.194:0/1070968834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 msgr2=0x7f24340f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.324+0000 7f2439216700 1 --2- 172.21.15.194:0/1070968834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f24340f6930 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7f2424009fb0 tx=0x7f2424009b30).stop 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 -- 172.21.15.194:0/1070968834 shutdown_connections 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 --2- 172.21.15.194:0/1070968834 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f24340f6930 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 -- 172.21.15.194:0/1070968834 >> 172.21.15.194:0/1070968834 conn(0x7f24340f1c70 msgr2=0x7f24340f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 -- 172.21.15.194:0/1070968834 shutdown_connections 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 -- 172.21.15.194:0/1070968834 wait complete. 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 Processor -- start 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 -- start start 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f2434106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.325+0000 7f2439216700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2434106c20 con 0x7f24340f6510 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f2432d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f2434106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:26.316 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f2432d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f2434106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48152/0 (socket says 172.21.15.194:48152) 2024-01-27T23:27:26.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f2432d9d700 1 -- 172.21.15.194:0/257276568 learned_addr learned my addr 172.21.15.194:0/257276568 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:26.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f2432d9d700 1 -- 172.21.15.194:0/257276568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2424005040 con 0x7f24340f6510 2024-01-27T23:27:26.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f2432d9d700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f2434106730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2424013510 tx=0x7f242400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f242b7fe700 1 -- 172.21.15.194:0/257276568 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2424009320 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2434106e20 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f242b7fe700 1 -- 172.21.15.194:0/257276568 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2424013f00 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f242b7fe700 1 -- 172.21.15.194:0/257276568 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2424014d90 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.326+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24341072b0 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.327+0000 7f242b7fe700 1 -- 172.21.15.194:0/257276568 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2424014420 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.327+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2434045ad0 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.327+0000 7f242b7fe700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f242002ff60 0x7f2420032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.327+0000 7f242b7fe700 1 -- 172.21.15.194:0/257276568 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2424041050 con 0x7f24340f6510 2024-01-27T23:27:26.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.328+0000 7f243259c700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f242002ff60 0x7f2420032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:26.319 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.329+0000 7f243259c700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f242002ff60 0x7f2420032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f241c0096a0 tx=0x7f241c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:26.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.332+0000 7f242b7fe700 1 -- 172.21.15.194:0/257276568 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f24240095f0 con 0x7f24340f6510 2024-01-27T23:27:26.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.537+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24340fbf60 con 0x7f24340f6510 2024-01-27T23:27:26.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.537+0000 7f242b7fe700 1 -- 172.21.15.194:0/257276568 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f242401db70 con 0x7f24340f6510 2024-01-27T23:27:26.527 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:26.527 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f242002ff60 msgr2=0x7f2420032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f242002ff60 0x7f2420032420 secure :-1 s=READY pgs=45 cs=0 l=1 rev1=1 rx=0x7f241c0096a0 tx=0x7f241c006b40).stop 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 msgr2=0x7f2434106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f2434106730 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f2424013510 tx=0x7f242400b040).stop 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 shutdown_connections 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f242002ff60 0x7f2420032420 unknown :-1 s=CLOSED pgs=45 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 --2- 172.21.15.194:0/257276568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24340f6510 0x7f2434106730 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 >> 172.21.15.194:0/257276568 conn(0x7f24340f1c70 msgr2=0x7f24340f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 shutdown_connections 2024-01-27T23:27:26.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:26.539+0000 7f2439216700 1 -- 172.21.15.194:0/257276568 wait complete. 2024-01-27T23:27:26.530 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:26.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:25.275377+0000 mgr.smithi121.hcppvz (mgr.14162) 65 : cluster [DBG] pgmap v58: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:27.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:26.538809+0000 mon.smithi121 (mon.0) 156 : audit [DBG] from='client.? 172.21.15.194:0/257276568' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:27.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:27.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:28.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:27.275640+0000 mgr.smithi121.hcppvz (mgr.14162) 66 : cluster [DBG] pgmap v59: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:29.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 -- 172.21.15.194:0/3468012724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 msgr2=0x7f0f840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:29.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 --2- 172.21.15.194:0/3468012724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f840f5830 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7f0f6c009fb0 tx=0x7f0f6c009b30).stop 2024-01-27T23:27:29.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 -- 172.21.15.194:0/3468012724 shutdown_connections 2024-01-27T23:27:29.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 --2- 172.21.15.194:0/3468012724 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f840f5830 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:29.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 -- 172.21.15.194:0/3468012724 >> 172.21.15.194:0/3468012724 conn(0x7f0f840f0b70 msgr2=0x7f0f840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 -- 172.21.15.194:0/3468012724 shutdown_connections 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 -- 172.21.15.194:0/3468012724 wait complete. 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 Processor -- start 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.068+0000 7f0f88de2700 1 -- start start 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f88de2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f84106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f88de2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f84106c10 con 0x7f0f840f5410 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f8259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f84106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f8259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f84106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48166/0 (socket says 172.21.15.194:48166) 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f8259c700 1 -- 172.21.15.194:0/870284053 learned_addr learned my addr 172.21.15.194:0/870284053 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f8259c700 1 -- 172.21.15.194:0/870284053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f6c005040 con 0x7f0f840f5410 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f8259c700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f84106720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0f6c013510 tx=0x7f0f6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f7affd700 1 -- 172.21.15.194:0/870284053 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f6c009320 con 0x7f0f840f5410 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f7affd700 1 -- 172.21.15.194:0/870284053 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f6c013f00 con 0x7f0f840f5410 2024-01-27T23:27:29.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f84106e10 con 0x7f0f840f5410 2024-01-27T23:27:29.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f7affd700 1 -- 172.21.15.194:0/870284053 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f6c014d90 con 0x7f0f840f5410 2024-01-27T23:27:29.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.069+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f841072a0 con 0x7f0f840f5410 2024-01-27T23:27:29.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.070+0000 7f0f7affd700 1 -- 172.21.15.194:0/870284053 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f6c014420 con 0x7f0f840f5410 2024-01-27T23:27:29.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.070+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f84045ad0 con 0x7f0f840f5410 2024-01-27T23:27:29.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.070+0000 7f0f7affd700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0f7002ff60 0x7f0f70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:29.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.070+0000 7f0f7affd700 1 -- 172.21.15.194:0/870284053 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f6c041000 con 0x7f0f840f5410 2024-01-27T23:27:29.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.071+0000 7f0f81d9b700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0f7002ff60 0x7f0f70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:29.062 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.071+0000 7f0f81d9b700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0f7002ff60 0x7f0f70032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f0f740096a0 tx=0x7f0f74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:29.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.075+0000 7f0f7affd700 1 -- 172.21.15.194:0/870284053 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f6c0095f0 con 0x7f0f840f5410 2024-01-27T23:27:29.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.281+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f84108150 con 0x7f0f840f5410 2024-01-27T23:27:29.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.282+0000 7f0f7affd700 1 -- 172.21.15.194:0/870284053 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f6c01db70 con 0x7f0f840f5410 2024-01-27T23:27:29.271 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:29.272 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:29.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.283+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0f7002ff60 msgr2=0x7f0f70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:29.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.283+0000 7f0f88de2700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0f7002ff60 0x7f0f70032420 secure :-1 s=READY pgs=46 cs=0 l=1 rev1=1 rx=0x7f0f740096a0 tx=0x7f0f74006b40).stop 2024-01-27T23:27:29.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.283+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 msgr2=0x7f0f84106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:29.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.283+0000 7f0f88de2700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f84106720 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f0f6c013510 tx=0x7f0f6c00b040).stop 2024-01-27T23:27:29.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.284+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 shutdown_connections 2024-01-27T23:27:29.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.284+0000 7f0f88de2700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0f7002ff60 0x7f0f70032420 unknown :-1 s=CLOSED pgs=46 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:29.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.284+0000 7f0f88de2700 1 --2- 172.21.15.194:0/870284053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f840f5410 0x7f0f84106720 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:29.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.284+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 >> 172.21.15.194:0/870284053 conn(0x7f0f840f0b70 msgr2=0x7f0f8405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:29.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.284+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 shutdown_connections 2024-01-27T23:27:29.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:29.284+0000 7f0f88de2700 1 -- 172.21.15.194:0/870284053 wait complete. 2024-01-27T23:27:29.274 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:29.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:29 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:29.283088+0000 mon.smithi121 (mon.0) 157 : audit [DBG] from='client.? 172.21.15.194:0/870284053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:30.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:29.275917+0000 mgr.smithi121.hcppvz (mgr.14162) 67 : cluster [DBG] pgmap v60: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:30.671 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:30.671 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:31.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.776+0000 7f75b9787700 1 -- 172.21.15.194:0/975547552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 msgr2=0x7f75b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.776+0000 7f75b9787700 1 --2- 172.21.15.194:0/975547552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b40f6930 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f759c009fb0 tx=0x7f759c009b30).stop 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.776+0000 7f75b9787700 1 -- 172.21.15.194:0/975547552 shutdown_connections 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.776+0000 7f75b9787700 1 --2- 172.21.15.194:0/975547552 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b40f6930 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.776+0000 7f75b9787700 1 -- 172.21.15.194:0/975547552 >> 172.21.15.194:0/975547552 conn(0x7f75b40f1c70 msgr2=0x7f75b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.776+0000 7f75b9787700 1 -- 172.21.15.194:0/975547552 shutdown_connections 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.776+0000 7f75b9787700 1 -- 172.21.15.194:0/975547552 wait complete. 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.777+0000 7f75b9787700 1 Processor -- start 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.777+0000 7f75b9787700 1 -- start start 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.777+0000 7f75b9787700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:31.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.777+0000 7f75b9787700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75b4106c20 con 0x7f75b40f6510 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.777+0000 7f75b2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.777+0000 7f75b2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48184/0 (socket says 172.21.15.194:48184) 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.777+0000 7f75b2ffd700 1 -- 172.21.15.194:0/1743550574 learned_addr learned my addr 172.21.15.194:0/1743550574 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.778+0000 7f75b2ffd700 1 -- 172.21.15.194:0/1743550574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f759c005040 con 0x7f75b40f6510 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.778+0000 7f75b2ffd700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b4106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f759c014000 tx=0x7f759c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.778+0000 7f75ab7fe700 1 -- 172.21.15.194:0/1743550574 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f759c0095f0 con 0x7f75b40f6510 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.778+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75b4106e20 con 0x7f75b40f6510 2024-01-27T23:27:31.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.778+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75b41072b0 con 0x7f75b40f6510 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.778+0000 7f75ab7fe700 1 -- 172.21.15.194:0/1743550574 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f759c007e70 con 0x7f75b40f6510 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.778+0000 7f75ab7fe700 1 -- 172.21.15.194:0/1743550574 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f759c014e50 con 0x7f75b40f6510 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.779+0000 7f75ab7fe700 1 -- 172.21.15.194:0/1743550574 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f759c025070 con 0x7f75b40f6510 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.779+0000 7f75ab7fe700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75a002ff10 0x7f75a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.779+0000 7f75ab7fe700 1 -- 172.21.15.194:0/1743550574 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f759c040ad0 con 0x7f75b40f6510 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.779+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75b4045ad0 con 0x7f75b40f6510 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.779+0000 7f75b27fc700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75a002ff10 0x7f75a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:31.770 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.780+0000 7f75b27fc700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75a002ff10 0x7f75a00323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f75a40096a0 tx=0x7f75a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:31.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.784+0000 7f75ab7fe700 1 -- 172.21.15.194:0/1743550574 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f759c026350 con 0x7f75b40f6510 2024-01-27T23:27:31.979 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.989+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75b40fc050 con 0x7f75b40f6510 2024-01-27T23:27:31.979 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.989+0000 7f75ab7fe700 1 -- 172.21.15.194:0/1743550574 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f759c012330 con 0x7f75b40f6510 2024-01-27T23:27:31.980 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:31.980 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75a002ff10 msgr2=0x7f75a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75a002ff10 0x7f75a00323d0 secure :-1 s=READY pgs=47 cs=0 l=1 rev1=1 rx=0x7f75a40096a0 tx=0x7f75a4006b40).stop 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 msgr2=0x7f75b4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b4106730 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f759c014000 tx=0x7f759c00b040).stop 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 shutdown_connections 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75a002ff10 0x7f75a00323d0 unknown :-1 s=CLOSED pgs=47 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 --2- 172.21.15.194:0/1743550574 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75b40f6510 0x7f75b4106730 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 >> 172.21.15.194:0/1743550574 conn(0x7f75b40f1c70 msgr2=0x7f75b40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 shutdown_connections 2024-01-27T23:27:31.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:31.991+0000 7f75b9787700 1 -- 172.21.15.194:0/1743550574 wait complete. 2024-01-27T23:27:31.982 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:32.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:31.276200+0000 mgr.smithi121.hcppvz (mgr.14162) 68 : cluster [DBG] pgmap v61: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:32.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:31.990691+0000 mon.smithi121 (mon.0) 158 : audit [DBG] from='client.? 172.21.15.194:0/1743550574' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:33.379 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:33.379 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:34.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.547+0000 7f3d3e435700 1 -- 172.21.15.194:0/544289506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 msgr2=0x7f3d380f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.547+0000 7f3d3e435700 1 --2- 172.21.15.194:0/544289506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380f5830 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f3d28009fb0 tx=0x7f3d28009b30).stop 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.547+0000 7f3d3e435700 1 -- 172.21.15.194:0/544289506 shutdown_connections 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.547+0000 7f3d3e435700 1 --2- 172.21.15.194:0/544289506 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380f5830 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.547+0000 7f3d3e435700 1 -- 172.21.15.194:0/544289506 >> 172.21.15.194:0/544289506 conn(0x7f3d380f0b70 msgr2=0x7f3d380f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.547+0000 7f3d3e435700 1 -- 172.21.15.194:0/544289506 shutdown_connections 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.547+0000 7f3d3e435700 1 -- 172.21.15.194:0/544289506 wait complete. 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.548+0000 7f3d3e435700 1 Processor -- start 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.548+0000 7f3d3e435700 1 -- start start 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.548+0000 7f3d3e435700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380fd330 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:34.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.548+0000 7f3d3e435700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3d380fb820 con 0x7f3d380f5410 2024-01-27T23:27:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.548+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380fd330 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.548+0000 7f3d37fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380fd330 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48188/0 (socket says 172.21.15.194:48188) 2024-01-27T23:27:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.548+0000 7f3d37fff700 1 -- 172.21.15.194:0/2227573680 learned_addr learned my addr 172.21.15.194:0/2227573680 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.549+0000 7f3d37fff700 1 -- 172.21.15.194:0/2227573680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3d28005040 con 0x7f3d380f5410 2024-01-27T23:27:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.549+0000 7f3d37fff700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380fd330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3d28013510 tx=0x7f3d2800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.549+0000 7f3d34ff9700 1 -- 172.21.15.194:0/2227573680 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d28009320 con 0x7f3d380f5410 2024-01-27T23:27:34.540 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.549+0000 7f3d34ff9700 1 -- 172.21.15.194:0/2227573680 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3d28013f00 con 0x7f3d380f5410 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.549+0000 7f3d34ff9700 1 -- 172.21.15.194:0/2227573680 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3d28014d90 con 0x7f3d380f5410 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.549+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3d380fba20 con 0x7f3d380f5410 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.549+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3d380fbeb0 con 0x7f3d380f5410 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.550+0000 7f3d34ff9700 1 -- 172.21.15.194:0/2227573680 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3d280263a0 con 0x7f3d380f5410 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.550+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3d38059100 con 0x7f3d380f5410 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.550+0000 7f3d34ff9700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3d2002ff60 0x7f3d20032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.550+0000 7f3d34ff9700 1 -- 172.21.15.194:0/2227573680 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3d280412b0 con 0x7f3d380f5410 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.550+0000 7f3d377fe700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3d2002ff60 0x7f3d20032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:34.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.551+0000 7f3d377fe700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3d2002ff60 0x7f3d20032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f3d2c0096a0 tx=0x7f3d2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:34.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.554+0000 7f3d34ff9700 1 -- 172.21.15.194:0/2227573680 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3d2801ccf0 con 0x7f3d380f5410 2024-01-27T23:27:34.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:33.276534+0000 mgr.smithi121.hcppvz (mgr.14162) 69 : cluster [DBG] pgmap v62: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:34.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.760+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3d380fae60 con 0x7f3d380f5410 2024-01-27T23:27:34.750 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.761+0000 7f3d34ff9700 1 -- 172.21.15.194:0/2227573680 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3d2801cb10 con 0x7f3d380f5410 2024-01-27T23:27:34.751 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:34.751 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:34.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3d2002ff60 msgr2=0x7f3d20032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:34.752 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3d2002ff60 0x7f3d20032420 secure :-1 s=READY pgs=48 cs=0 l=1 rev1=1 rx=0x7f3d2c0096a0 tx=0x7f3d2c006b40).stop 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 msgr2=0x7f3d380fd330 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380fd330 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7f3d28013510 tx=0x7f3d2800b040).stop 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 shutdown_connections 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3d2002ff60 0x7f3d20032420 unknown :-1 s=CLOSED pgs=48 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 --2- 172.21.15.194:0/2227573680 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3d380f5410 0x7f3d380fd330 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 >> 172.21.15.194:0/2227573680 conn(0x7f3d380f0b70 msgr2=0x7f3d3805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 shutdown_connections 2024-01-27T23:27:34.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:34.762+0000 7f3d3e435700 1 -- 172.21.15.194:0/2227573680 wait complete. 2024-01-27T23:27:34.754 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:35.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:35 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:34.762201+0000 mon.smithi121 (mon.0) 159 : audit [DBG] from='client.? 172.21.15.194:0/2227573680' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:36.078 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:36.079 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:36.586 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:35.276810+0000 mgr.smithi121.hcppvz (mgr.14162) 70 : cluster [DBG] pgmap v63: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:37.271 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.280+0000 7fd8d3225700 1 -- 172.21.15.194:0/1318825144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 msgr2=0x7fd8cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.280+0000 7fd8d3225700 1 --2- 172.21.15.194:0/1318825144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc0f5830 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7fd8c0009fb0 tx=0x7fd8c0009b30).stop 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.280+0000 7fd8d3225700 1 -- 172.21.15.194:0/1318825144 shutdown_connections 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.280+0000 7fd8d3225700 1 --2- 172.21.15.194:0/1318825144 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc0f5830 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.280+0000 7fd8d3225700 1 -- 172.21.15.194:0/1318825144 >> 172.21.15.194:0/1318825144 conn(0x7fd8cc0f0b70 msgr2=0x7fd8cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.280+0000 7fd8d3225700 1 -- 172.21.15.194:0/1318825144 shutdown_connections 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.280+0000 7fd8d3225700 1 -- 172.21.15.194:0/1318825144 wait complete. 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d3225700 1 Processor -- start 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d3225700 1 -- start start 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d3225700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc1093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d3225700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd8cc060280 con 0x7fd8cc0f5410 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d0fc1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc1093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:37.272 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d0fc1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc1093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48208/0 (socket says 172.21.15.194:48208) 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d0fc1700 1 -- 172.21.15.194:0/2732794459 learned_addr learned my addr 172.21.15.194:0/2732794459 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.281+0000 7fd8d0fc1700 1 -- 172.21.15.194:0/2732794459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd8c0005040 con 0x7fd8cc0f5410 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.282+0000 7fd8d0fc1700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc1093f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd8c0013510 tx=0x7fd8c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.282+0000 7fd8c97fa700 1 -- 172.21.15.194:0/2732794459 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8c0009320 con 0x7fd8cc0f5410 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.282+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd8cc060480 con 0x7fd8cc0f5410 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.282+0000 7fd8c97fa700 1 -- 172.21.15.194:0/2732794459 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd8c0013f00 con 0x7fd8cc0f5410 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.282+0000 7fd8c97fa700 1 -- 172.21.15.194:0/2732794459 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd8c0014d90 con 0x7fd8cc0f5410 2024-01-27T23:27:37.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.282+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd8cc109b20 con 0x7fd8cc0f5410 2024-01-27T23:27:37.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.283+0000 7fd8c97fa700 1 -- 172.21.15.194:0/2732794459 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd8c0014420 con 0x7fd8cc0f5410 2024-01-27T23:27:37.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.283+0000 7fd8c97fa700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd8bc02ffb0 0x7fd8bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:37.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.283+0000 7fd8c97fa700 1 -- 172.21.15.194:0/2732794459 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd8c00410c0 con 0x7fd8cc0f5410 2024-01-27T23:27:37.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.283+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd8cc045ad0 con 0x7fd8cc0f5410 2024-01-27T23:27:37.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.283+0000 7fd8cbfff700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd8bc02ffb0 0x7fd8bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:37.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.284+0000 7fd8cbfff700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd8bc02ffb0 0x7fd8bc032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd8b80096a0 tx=0x7fd8b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:37.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.288+0000 7fd8c97fa700 1 -- 172.21.15.194:0/2732794459 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd8c00095f0 con 0x7fd8cc0f5410 2024-01-27T23:27:37.482 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.492+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd8cc10a1d0 con 0x7fd8cc0f5410 2024-01-27T23:27:37.483 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.493+0000 7fd8c97fa700 1 -- 172.21.15.194:0/2732794459 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd8c001db70 con 0x7fd8cc0f5410 2024-01-27T23:27:37.483 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:37.483 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:37.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd8bc02ffb0 msgr2=0x7fd8bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:37.484 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd8bc02ffb0 0x7fd8bc032470 secure :-1 s=READY pgs=49 cs=0 l=1 rev1=1 rx=0x7fd8b80096a0 tx=0x7fd8b8006b40).stop 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 msgr2=0x7fd8cc1093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc1093f0 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7fd8c0013510 tx=0x7fd8c000b040).stop 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 shutdown_connections 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd8bc02ffb0 0x7fd8bc032470 unknown :-1 s=CLOSED pgs=49 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 --2- 172.21.15.194:0/2732794459 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd8cc0f5410 0x7fd8cc1093f0 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 >> 172.21.15.194:0/2732794459 conn(0x7fd8cc0f0b70 msgr2=0x7fd8cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 shutdown_connections 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:37.495+0000 7fd8d3225700 1 -- 172.21.15.194:0/2732794459 wait complete. 2024-01-27T23:27:37.485 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:38.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:37.277029+0000 mgr.smithi121.hcppvz (mgr.14162) 71 : cluster [DBG] pgmap v64: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:38.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:37.494553+0000 mon.smithi121 (mon.0) 160 : audit [DBG] from='client.? 172.21.15.194:0/2732794459' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:38.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:38.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.082+0000 7fb681c96700 1 -- 172.21.15.194:0/456957909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 msgr2=0x7fb67c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.082+0000 7fb681c96700 1 --2- 172.21.15.194:0/456957909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c0f5830 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7fb670009fb0 tx=0x7fb670009b30).stop 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.082+0000 7fb681c96700 1 -- 172.21.15.194:0/456957909 shutdown_connections 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.082+0000 7fb681c96700 1 --2- 172.21.15.194:0/456957909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c0f5830 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.082+0000 7fb681c96700 1 -- 172.21.15.194:0/456957909 >> 172.21.15.194:0/456957909 conn(0x7fb67c0f0b70 msgr2=0x7fb67c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.082+0000 7fb681c96700 1 -- 172.21.15.194:0/456957909 shutdown_connections 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.082+0000 7fb681c96700 1 -- 172.21.15.194:0/456957909 wait complete. 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.083+0000 7fb681c96700 1 Processor -- start 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.083+0000 7fb681c96700 1 -- start start 2024-01-27T23:27:40.074 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.083+0000 7fb681c96700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.083+0000 7fb681c96700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb67c106c50 con 0x7fb67c0f5410 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.083+0000 7fb67b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.083+0000 7fb67b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48234/0 (socket says 172.21.15.194:48234) 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.083+0000 7fb67b7fe700 1 -- 172.21.15.194:0/1277684039 learned_addr learned my addr 172.21.15.194:0/1277684039 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.084+0000 7fb67b7fe700 1 -- 172.21.15.194:0/1277684039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb670005040 con 0x7fb67c0f5410 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.084+0000 7fb67b7fe700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c106760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb670009f80 tx=0x7fb67000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.084+0000 7fb663fff700 1 -- 172.21.15.194:0/1277684039 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb670009320 con 0x7fb67c0f5410 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.084+0000 7fb663fff700 1 -- 172.21.15.194:0/1277684039 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb670013ec0 con 0x7fb67c0f5410 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.084+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb67c106e50 con 0x7fb67c0f5410 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.084+0000 7fb663fff700 1 -- 172.21.15.194:0/1277684039 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb67001ece0 con 0x7fb67c0f5410 2024-01-27T23:27:40.075 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.084+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb67c1072e0 con 0x7fb67c0f5410 2024-01-27T23:27:40.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.085+0000 7fb663fff700 1 -- 172.21.15.194:0/1277684039 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb670014420 con 0x7fb67c0f5410 2024-01-27T23:27:40.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.085+0000 7fb663fff700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb66402ff60 0x7fb664032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:40.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.085+0000 7fb663fff700 1 -- 172.21.15.194:0/1277684039 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb670040dd0 con 0x7fb67c0f5410 2024-01-27T23:27:40.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.085+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb67c045ad0 con 0x7fb67c0f5410 2024-01-27T23:27:40.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.085+0000 7fb67affd700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb66402ff60 0x7fb664032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:40.077 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.085+0000 7fb67affd700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb66402ff60 0x7fb664032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb66c0096a0 tx=0x7fb66c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:40.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.090+0000 7fb663fff700 1 -- 172.21.15.194:0/1277684039 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb67001c910 con 0x7fb67c0f5410 2024-01-27T23:27:40.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.298+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb67c108030 con 0x7fb67c0f5410 2024-01-27T23:27:40.288 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.298+0000 7fb663fff700 1 -- 172.21.15.194:0/1277684039 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb67001c320 con 0x7fb67c0f5410 2024-01-27T23:27:40.288 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:40.288 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb66402ff60 msgr2=0x7fb664032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb66402ff60 0x7fb664032420 secure :-1 s=READY pgs=50 cs=0 l=1 rev1=1 rx=0x7fb66c0096a0 tx=0x7fb66c006b40).stop 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 msgr2=0x7fb67c106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c106760 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fb670009f80 tx=0x7fb67000b040).stop 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 shutdown_connections 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb66402ff60 0x7fb664032420 unknown :-1 s=CLOSED pgs=50 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 --2- 172.21.15.194:0/1277684039 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb67c0f5410 0x7fb67c106760 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 >> 172.21.15.194:0/1277684039 conn(0x7fb67c0f0b70 msgr2=0x7fb67c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 shutdown_connections 2024-01-27T23:27:40.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:40.300+0000 7fb681c96700 1 -- 172.21.15.194:0/1277684039 wait complete. 2024-01-27T23:27:40.291 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:40.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:39.277297+0000 mgr.smithi121.hcppvz (mgr.14162) 72 : cluster [DBG] pgmap v65: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:40.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:40.299572+0000 mon.smithi121 (mon.0) 161 : audit [DBG] from='client.? 172.21.15.194:0/1277684039' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:41.670 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:41.670 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:42.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:41.277531+0000 mgr.smithi121.hcppvz (mgr.14162) 73 : cluster [DBG] pgmap v66: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:42.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.829+0000 7f501060f700 1 -- 172.21.15.194:0/2732185401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 msgr2=0x7f50080f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:42.821 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.829+0000 7f501060f700 1 --2- 172.21.15.194:0/2732185401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f50080f5830 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7f4ff8009fb0 tx=0x7f4ff8009b30).stop 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.829+0000 7f501060f700 1 -- 172.21.15.194:0/2732185401 shutdown_connections 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.829+0000 7f501060f700 1 --2- 172.21.15.194:0/2732185401 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f50080f5830 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.829+0000 7f501060f700 1 -- 172.21.15.194:0/2732185401 >> 172.21.15.194:0/2732185401 conn(0x7f50080f0b70 msgr2=0x7f50080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.829+0000 7f501060f700 1 -- 172.21.15.194:0/2732185401 shutdown_connections 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.829+0000 7f501060f700 1 -- 172.21.15.194:0/2732185401 wait complete. 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f501060f700 1 Processor -- start 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f501060f700 1 -- start start 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f501060f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f500810b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f501060f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5008107240 con 0x7f50080f5410 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f500e3ab700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f500810b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f500e3ab700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f500810b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48248/0 (socket says 172.21.15.194:48248) 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f500e3ab700 1 -- 172.21.15.194:0/1375885364 learned_addr learned my addr 172.21.15.194:0/1375885364 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:42.822 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f500e3ab700 1 -- 172.21.15.194:0/1375885364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ff8005040 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.830+0000 7f500e3ab700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f500810b850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4ff8014000 tx=0x7f4ff800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.831+0000 7f4ffeffd700 1 -- 172.21.15.194:0/1375885364 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff80095f0 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.831+0000 7f4ffeffd700 1 -- 172.21.15.194:0/1375885364 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ff8007e70 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.831+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5008060220 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.831+0000 7f4ffeffd700 1 -- 172.21.15.194:0/1375885364 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ff8014ea0 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.831+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f500810bd40 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.831+0000 7f4ffeffd700 1 -- 172.21.15.194:0/1375885364 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ff8025070 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.831+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5008045ad0 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.832+0000 7f4ffeffd700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ff402ffb0 0x7f4ff4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.832+0000 7f4ffeffd700 1 -- 172.21.15.194:0/1375885364 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ff80409a0 con 0x7f50080f5410 2024-01-27T23:27:42.823 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.832+0000 7f500dbaa700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ff402ffb0 0x7f4ff4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:42.824 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.832+0000 7f500dbaa700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ff402ffb0 0x7f4ff4032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f50040096a0 tx=0x7f5004006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:42.826 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:42.836+0000 7f4ffeffd700 1 -- 172.21.15.194:0/1375885364 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ff80079c0 con 0x7f50080f5410 2024-01-27T23:27:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.044+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f50080fae60 con 0x7f50080f5410 2024-01-27T23:27:43.035 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.044+0000 7f4ffeffd700 1 -- 172.21.15.194:0/1375885364 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4ff800f300 con 0x7f50080f5410 2024-01-27T23:27:43.036 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:43.036 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ff402ffb0 msgr2=0x7f4ff4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ff402ffb0 0x7f4ff4032470 secure :-1 s=READY pgs=51 cs=0 l=1 rev1=1 rx=0x7f50040096a0 tx=0x7f5004006b40).stop 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 msgr2=0x7f500810b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f500810b850 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f4ff8014000 tx=0x7f4ff800b040).stop 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 shutdown_connections 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ff402ffb0 0x7f4ff4032470 unknown :-1 s=CLOSED pgs=51 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 --2- 172.21.15.194:0/1375885364 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f50080f5410 0x7f500810b850 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 >> 172.21.15.194:0/1375885364 conn(0x7f50080f0b70 msgr2=0x7f50080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 shutdown_connections 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:43.046+0000 7f501060f700 1 -- 172.21.15.194:0/1375885364 wait complete. 2024-01-27T23:27:43.037 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:43.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:43.045754+0000 mon.smithi121 (mon.0) 162 : audit [DBG] from='client.? 172.21.15.194:0/1375885364' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:44.401 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:44.401 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:44.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:43.277835+0000 mgr.smithi121.hcppvz (mgr.14162) 74 : cluster [DBG] pgmap v67: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:45.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.591+0000 7f579c82b700 1 -- 172.21.15.194:0/3383790203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 msgr2=0x7f57980f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:45.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.591+0000 7f579c82b700 1 --2- 172.21.15.194:0/3383790203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57980f66c0 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f5788009fb0 tx=0x7f5788009b30).stop 2024-01-27T23:27:45.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.591+0000 7f579c82b700 1 -- 172.21.15.194:0/3383790203 shutdown_connections 2024-01-27T23:27:45.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.591+0000 7f579c82b700 1 --2- 172.21.15.194:0/3383790203 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57980f66c0 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:45.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.591+0000 7f579c82b700 1 -- 172.21.15.194:0/3383790203 >> 172.21.15.194:0/3383790203 conn(0x7f57980f1aa0 msgr2=0x7f57980f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:45.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.591+0000 7f579c82b700 1 -- 172.21.15.194:0/3383790203 shutdown_connections 2024-01-27T23:27:45.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.591+0000 7f579c82b700 1 -- 172.21.15.194:0/3383790203 wait complete. 2024-01-27T23:27:45.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.592+0000 7f579c82b700 1 Processor -- start 2024-01-27T23:27:45.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.592+0000 7f579c82b700 1 -- start start 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f579c82b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57981064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f579c82b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57981069a0 con 0x7f57980f62a0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f5796d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57981064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f5796d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57981064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:38596/0 (socket says 172.21.15.194:38596) 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f5796d9d700 1 -- 172.21.15.194:0/3380641078 learned_addr learned my addr 172.21.15.194:0/3380641078 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f5796d9d700 1 -- 172.21.15.194:0/3380641078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5788005040 con 0x7f57980f62a0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f5796d9d700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57981064b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5788013510 tx=0x7f578800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f577f7fe700 1 -- 172.21.15.194:0/3380641078 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5788009320 con 0x7f57980f62a0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f577f7fe700 1 -- 172.21.15.194:0/3380641078 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5788013f00 con 0x7f57980f62a0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5798106ba0 con 0x7f57980f62a0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f577f7fe700 1 -- 172.21.15.194:0/3380641078 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5788014dc0 con 0x7f57980f62a0 2024-01-27T23:27:45.584 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.593+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5798107030 con 0x7f57980f62a0 2024-01-27T23:27:45.585 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.594+0000 7f577f7fe700 1 -- 172.21.15.194:0/3380641078 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f578801d3d0 con 0x7f57980f62a0 2024-01-27T23:27:45.585 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.594+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5798045ad0 con 0x7f57980f62a0 2024-01-27T23:27:45.585 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.594+0000 7f577f7fe700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f578002ff60 0x7f5780032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:45.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.594+0000 7f577f7fe700 1 -- 172.21.15.194:0/3380641078 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5788041430 con 0x7f57980f62a0 2024-01-27T23:27:45.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.594+0000 7f579659c700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f578002ff60 0x7f5780032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:45.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.595+0000 7f579659c700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f578002ff60 0x7f5780032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:45.588 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.599+0000 7f577f7fe700 1 -- 172.21.15.194:0/3380641078 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5788012330 con 0x7f57980f62a0 2024-01-27T23:27:45.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.805+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5798107ee0 con 0x7f57980f62a0 2024-01-27T23:27:45.795 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.805+0000 7f577f7fe700 1 -- 172.21.15.194:0/3380641078 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5788012330 con 0x7f57980f62a0 2024-01-27T23:27:45.795 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:45.795 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:45.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.806+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f578002ff60 msgr2=0x7f5780032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:45.796 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.806+0000 7f579c82b700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f578002ff60 0x7f5780032420 secure :-1 s=READY pgs=52 cs=0 l=1 rev1=1 rx=0x7f578c0096a0 tx=0x7f578c006b40).stop 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.806+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 msgr2=0x7f57981064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.806+0000 7f579c82b700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57981064b0 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f5788013510 tx=0x7f578800b040).stop 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.806+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 shutdown_connections 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.806+0000 7f579c82b700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f578002ff60 0x7f5780032420 unknown :-1 s=CLOSED pgs=52 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.807+0000 7f579c82b700 1 --2- 172.21.15.194:0/3380641078 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57980f62a0 0x7f57981064b0 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.807+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 >> 172.21.15.194:0/3380641078 conn(0x7f57980f1aa0 msgr2=0x7f57980f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.807+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 shutdown_connections 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:45.807+0000 7f579c82b700 1 -- 172.21.15.194:0/3380641078 wait complete. 2024-01-27T23:27:45.797 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:46.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:45.278152+0000 mgr.smithi121.hcppvz (mgr.14162) 75 : cluster [DBG] pgmap v68: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:46.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:45.806484+0000 mon.smithi121 (mon.0) 163 : audit [DBG] from='client.? 172.21.15.194:0/3380641078' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:47.241 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:47.241 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:48.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.415+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/3890285091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 msgr2=0x7fa7b40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:48.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.415+0000 7fa7bc2bf700 1 --2- 172.21.15.194:0/3890285091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b40f5830 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7fa7a8009fb0 tx=0x7fa7a8009b30).stop 2024-01-27T23:27:48.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.415+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/3890285091 shutdown_connections 2024-01-27T23:27:48.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.415+0000 7fa7bc2bf700 1 --2- 172.21.15.194:0/3890285091 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b40f5830 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:48.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.415+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/3890285091 >> 172.21.15.194:0/3890285091 conn(0x7fa7b40f0b70 msgr2=0x7fa7b40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:48.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.415+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/3890285091 shutdown_connections 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.415+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/3890285091 wait complete. 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7bc2bf700 1 Processor -- start 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7bc2bf700 1 -- start start 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7bc2bf700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7bc2bf700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa7b4105b20 con 0x7fa7b40f5410 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7ba05b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7ba05b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:38610/0 (socket says 172.21.15.194:38610) 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7ba05b700 1 -- 172.21.15.194:0/2371634808 learned_addr learned my addr 172.21.15.194:0/2371634808 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.416+0000 7fa7ba05b700 1 -- 172.21.15.194:0/2371634808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa7a8005040 con 0x7fa7b40f5410 2024-01-27T23:27:48.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.417+0000 7fa7ba05b700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b4105630 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa7a8013510 tx=0x7fa7a800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:48.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.417+0000 7fa7a6ffd700 1 -- 172.21.15.194:0/2371634808 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a8009320 con 0x7fa7b40f5410 2024-01-27T23:27:48.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.417+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa7b4105d20 con 0x7fa7b40f5410 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.417+0000 7fa7a6ffd700 1 -- 172.21.15.194:0/2371634808 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa7a8013f00 con 0x7fa7b40f5410 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.417+0000 7fa7a6ffd700 1 -- 172.21.15.194:0/2371634808 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa7a8014d90 con 0x7fa7b40f5410 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.417+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa7b41061b0 con 0x7fa7b40f5410 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.417+0000 7fa7a6ffd700 1 -- 172.21.15.194:0/2371634808 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa7a80095f0 con 0x7fa7b40f5410 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.418+0000 7fa7a6ffd700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa7a002ff60 0x7fa7a0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.418+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa7b4045ad0 con 0x7fa7b40f5410 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.418+0000 7fa7a6ffd700 1 -- 172.21.15.194:0/2371634808 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa7a8041030 con 0x7fa7b40f5410 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.418+0000 7fa7b985a700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa7a002ff60 0x7fa7a0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:48.409 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.418+0000 7fa7b985a700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa7a002ff60 0x7fa7a0032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa7b00096a0 tx=0x7fa7b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:48.412 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.422+0000 7fa7a6ffd700 1 -- 172.21.15.194:0/2371634808 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa7a801d7f0 con 0x7fa7b40f5410 2024-01-27T23:27:48.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:47.278440+0000 mgr.smithi121.hcppvz (mgr.14162) 76 : cluster [DBG] pgmap v69: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:48.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.627+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa7b40faf50 con 0x7fa7b40f5410 2024-01-27T23:27:48.617 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.628+0000 7fa7a6ffd700 1 -- 172.21.15.194:0/2371634808 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa7a8010230 con 0x7fa7b40f5410 2024-01-27T23:27:48.618 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:48.618 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa7a002ff60 msgr2=0x7fa7a0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa7a002ff60 0x7fa7a0032420 secure :-1 s=READY pgs=53 cs=0 l=1 rev1=1 rx=0x7fa7b00096a0 tx=0x7fa7b0006b40).stop 2024-01-27T23:27:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 msgr2=0x7fa7b4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b4105630 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fa7a8013510 tx=0x7fa7a800b040).stop 2024-01-27T23:27:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 shutdown_connections 2024-01-27T23:27:48.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa7a002ff60 0x7fa7a0032420 unknown :-1 s=CLOSED pgs=53 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 --2- 172.21.15.194:0/2371634808 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa7b40f5410 0x7fa7b4105630 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 >> 172.21.15.194:0/2371634808 conn(0x7fa7b40f0b70 msgr2=0x7fa7b40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 shutdown_connections 2024-01-27T23:27:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:48.629+0000 7fa7bc2bf700 1 -- 172.21.15.194:0/2371634808 wait complete. 2024-01-27T23:27:48.620 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:49.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:49 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:48.628969+0000 mon.smithi121 (mon.0) 164 : audit [DBG] from='client.? 172.21.15.194:0/2371634808' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:49.990 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:49.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:50.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:49.278829+0000 mgr.smithi121.hcppvz (mgr.14162) 77 : cluster [DBG] pgmap v70: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.186+0000 7f4d335a8700 1 -- 172.21.15.194:0/2150602396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 msgr2=0x7f4d2c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.186+0000 7f4d335a8700 1 --2- 172.21.15.194:0/2150602396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c0f6930 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f4d24009960 tx=0x7f4d24004d10).stop 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.186+0000 7f4d335a8700 1 -- 172.21.15.194:0/2150602396 shutdown_connections 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.186+0000 7f4d335a8700 1 --2- 172.21.15.194:0/2150602396 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c0f6930 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.186+0000 7f4d335a8700 1 -- 172.21.15.194:0/2150602396 >> 172.21.15.194:0/2150602396 conn(0x7f4d2c0f1c70 msgr2=0x7f4d2c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.186+0000 7f4d335a8700 1 -- 172.21.15.194:0/2150602396 shutdown_connections 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.186+0000 7f4d335a8700 1 -- 172.21.15.194:0/2150602396 wait complete. 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d335a8700 1 Processor -- start 2024-01-27T23:27:51.178 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d335a8700 1 -- start start 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d335a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d335a8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d2c069e20 con 0x7f4d2c0f6510 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d31344700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d31344700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:38630/0 (socket says 172.21.15.194:38630) 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d31344700 1 -- 172.21.15.194:0/115434085 learned_addr learned my addr 172.21.15.194:0/115434085 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.187+0000 7f4d31344700 1 -- 172.21.15.194:0/115434085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d24009640 con 0x7f4d2c0f6510 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.188+0000 7f4d31344700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c06b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f4d24004b70 tx=0x7f4d24004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.188+0000 7f4d21ffb700 1 -- 172.21.15.194:0/115434085 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d24018070 con 0x7f4d2c0f6510 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.188+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d2c069fc0 con 0x7f4d2c0f6510 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.188+0000 7f4d21ffb700 1 -- 172.21.15.194:0/115434085 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d2400b870 con 0x7f4d2c0f6510 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.188+0000 7f4d21ffb700 1 -- 172.21.15.194:0/115434085 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d2401c560 con 0x7f4d2c0f6510 2024-01-27T23:27:51.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.188+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d2c06a450 con 0x7f4d2c0f6510 2024-01-27T23:27:51.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.189+0000 7f4d21ffb700 1 -- 172.21.15.194:0/115434085 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d24013630 con 0x7f4d2c0f6510 2024-01-27T23:27:51.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.189+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d2c059100 con 0x7f4d2c0f6510 2024-01-27T23:27:51.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.189+0000 7f4d21ffb700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d1802ff60 0x7f4d18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:51.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.189+0000 7f4d21ffb700 1 -- 172.21.15.194:0/115434085 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d24041330 con 0x7f4d2c0f6510 2024-01-27T23:27:51.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.189+0000 7f4d30b43700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d1802ff60 0x7f4d18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:51.180 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.190+0000 7f4d30b43700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d1802ff60 0x7f4d18032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f4d1c0096a0 tx=0x7f4d1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:51.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.194+0000 7f4d21ffb700 1 -- 172.21.15.194:0/115434085 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d24004810 con 0x7f4d2c0f6510 2024-01-27T23:27:51.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.402+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d2c045ad0 con 0x7f4d2c0f6510 2024-01-27T23:27:51.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.403+0000 7f4d21ffb700 1 -- 172.21.15.194:0/115434085 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d24012380 con 0x7f4d2c0f6510 2024-01-27T23:27:51.394 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:51.394 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:51.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d1802ff60 msgr2=0x7f4d18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:51.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d1802ff60 0x7f4d18032420 secure :-1 s=READY pgs=54 cs=0 l=1 rev1=1 rx=0x7f4d1c0096a0 tx=0x7f4d1c006b40).stop 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 msgr2=0x7f4d2c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c06b9a0 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f4d24004b70 tx=0x7f4d24004d10).stop 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 shutdown_connections 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d1802ff60 0x7f4d18032420 unknown :-1 s=CLOSED pgs=54 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 --2- 172.21.15.194:0/115434085 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d2c0f6510 0x7f4d2c06b9a0 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 >> 172.21.15.194:0/115434085 conn(0x7f4d2c0f1c70 msgr2=0x7f4d2c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 shutdown_connections 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:51.405+0000 7f4d335a8700 1 -- 172.21.15.194:0/115434085 wait complete. 2024-01-27T23:27:51.396 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:52.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:51.279241+0000 mgr.smithi121.hcppvz (mgr.14162) 78 : cluster [DBG] pgmap v71: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:52.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:51.404567+0000 mon.smithi121 (mon.0) 165 : audit [DBG] from='client.? 172.21.15.194:0/115434085' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:52.787 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:52.787 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:54.010 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.020+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3522114328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 msgr2=0x7f66d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:54.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.020+0000 7f66dd1ca700 1 --2- 172.21.15.194:0/3522114328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d80f5830 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f66c0009fb0 tx=0x7f66c0009b30).stop 2024-01-27T23:27:54.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.020+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3522114328 shutdown_connections 2024-01-27T23:27:54.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.020+0000 7f66dd1ca700 1 --2- 172.21.15.194:0/3522114328 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d80f5830 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:54.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.020+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3522114328 >> 172.21.15.194:0/3522114328 conn(0x7f66d80f0b70 msgr2=0x7f66d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:54.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.020+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3522114328 shutdown_connections 2024-01-27T23:27:54.011 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.020+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3522114328 wait complete. 2024-01-27T23:27:54.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.021+0000 7f66dd1ca700 1 Processor -- start 2024-01-27T23:27:54.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.021+0000 7f66dd1ca700 1 -- start start 2024-01-27T23:27:54.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.021+0000 7f66dd1ca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:54.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.021+0000 7f66dd1ca700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66d8060240 con 0x7f66d80f5410 2024-01-27T23:27:54.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66d6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:54.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66d6d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:38642/0 (socket says 172.21.15.194:38642) 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66d6d9d700 1 -- 172.21.15.194:0/3278129041 learned_addr learned my addr 172.21.15.194:0/3278129041 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66d6d9d700 1 -- 172.21.15.194:0/3278129041 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f66c0005040 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66d6d9d700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d8109430 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f66c0014000 tx=0x7f66c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66cf7fe700 1 -- 172.21.15.194:0/3278129041 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c00095f0 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66d8060440 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66cf7fe700 1 -- 172.21.15.194:0/3278129041 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f66c0007e70 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66cf7fe700 1 -- 172.21.15.194:0/3278129041 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f66c0014ea0 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.022+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66d8109ae0 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.023+0000 7f66cf7fe700 1 -- 172.21.15.194:0/3278129041 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66c0025070 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.023+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f66d8045ad0 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.023+0000 7f66cf7fe700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.023+0000 7f66cf7fe700 1 -- 172.21.15.194:0/3278129041 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f66c0040970 con 0x7f66d80f5410 2024-01-27T23:27:54.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.023+0000 7f66d659c700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:54.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.024+0000 7f66d659c700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66c402ff60 0x7f66c4032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f66c80096a0 tx=0x7f66c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:54.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.028+0000 7f66cf7fe700 1 -- 172.21.15.194:0/3278129041 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f66c00079c0 con 0x7f66d80f5410 2024-01-27T23:27:54.223 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.233+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66d80fae60 con 0x7f66d80f5410 2024-01-27T23:27:54.223 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.234+0000 7f66cf7fe700 1 -- 172.21.15.194:0/3278129041 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f66c000f300 con 0x7f66d80f5410 2024-01-27T23:27:54.224 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:54.224 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:54.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66c402ff60 msgr2=0x7f66c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:54.225 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66c402ff60 0x7f66c4032420 secure :-1 s=READY pgs=55 cs=0 l=1 rev1=1 rx=0x7f66c80096a0 tx=0x7f66c8006b40).stop 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 msgr2=0x7f66d8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d8109430 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7f66c0014000 tx=0x7f66c000b040).stop 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 shutdown_connections 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66c402ff60 0x7f66c4032420 unknown :-1 s=CLOSED pgs=55 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 --2- 172.21.15.194:0/3278129041 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66d80f5410 0x7f66d8109430 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 >> 172.21.15.194:0/3278129041 conn(0x7f66d80f0b70 msgr2=0x7f66d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 shutdown_connections 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:54.235+0000 7f66dd1ca700 1 -- 172.21.15.194:0/3278129041 wait complete. 2024-01-27T23:27:54.226 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:54.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:53.279659+0000 mgr.smithi121.hcppvz (mgr.14162) 79 : cluster [DBG] pgmap v72: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:54.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:54.235150+0000 mon.smithi121 (mon.0) 166 : audit [DBG] from='client.? 172.21.15.194:0/3278129041' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:55.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:55.603 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:56.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:55.280114+0000 mgr.smithi121.hcppvz (mgr.14162) 80 : cluster [DBG] pgmap v73: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:56.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.857+0000 7fe84afe1700 1 -- 172.21.15.194:0/4294016066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 msgr2=0x7fe8440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:56.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.857+0000 7fe84afe1700 1 --2- 172.21.15.194:0/4294016066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe8440f5830 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fe838009fb0 tx=0x7fe838009b30).stop 2024-01-27T23:27:56.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.857+0000 7fe84afe1700 1 -- 172.21.15.194:0/4294016066 shutdown_connections 2024-01-27T23:27:56.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.857+0000 7fe84afe1700 1 --2- 172.21.15.194:0/4294016066 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe8440f5830 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:56.848 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.857+0000 7fe84afe1700 1 -- 172.21.15.194:0/4294016066 >> 172.21.15.194:0/4294016066 conn(0x7fe8440f0b70 msgr2=0x7fe8440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:56.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.857+0000 7fe84afe1700 1 -- 172.21.15.194:0/4294016066 shutdown_connections 2024-01-27T23:27:56.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.857+0000 7fe84afe1700 1 -- 172.21.15.194:0/4294016066 wait complete. 2024-01-27T23:27:56.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.858+0000 7fe84afe1700 1 Processor -- start 2024-01-27T23:27:56.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.858+0000 7fe84afe1700 1 -- start start 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe84afe1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe844105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe84afe1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe844105b20 con 0x7fe8440f5410 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe848d7d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe844105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe848d7d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe844105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:36242/0 (socket says 172.21.15.194:36242) 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe848d7d700 1 -- 172.21.15.194:0/599732344 learned_addr learned my addr 172.21.15.194:0/599732344 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe848d7d700 1 -- 172.21.15.194:0/599732344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe838005040 con 0x7fe8440f5410 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe848d7d700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe844105630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe838013510 tx=0x7fe83800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe8417fa700 1 -- 172.21.15.194:0/599732344 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe838009320 con 0x7fe8440f5410 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe844105d20 con 0x7fe8440f5410 2024-01-27T23:27:56.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe8417fa700 1 -- 172.21.15.194:0/599732344 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe838013f00 con 0x7fe8440f5410 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe8417fa700 1 -- 172.21.15.194:0/599732344 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe838014d90 con 0x7fe8440f5410 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.859+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe8441061b0 con 0x7fe8440f5410 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.860+0000 7fe8417fa700 1 -- 172.21.15.194:0/599732344 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe8380095f0 con 0x7fe8440f5410 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.860+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe844045ad0 con 0x7fe8440f5410 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.860+0000 7fe8417fa700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe83402ff60 0x7fe834032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.860+0000 7fe8417fa700 1 -- 172.21.15.194:0/599732344 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe838041020 con 0x7fe8440f5410 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.861+0000 7fe843fff700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe83402ff60 0x7fe834032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:56.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.861+0000 7fe843fff700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe83402ff60 0x7fe834032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe8300096a0 tx=0x7fe830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:56.855 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:56.865+0000 7fe8417fa700 1 -- 172.21.15.194:0/599732344 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe838014420 con 0x7fe8440f5410 2024-01-27T23:27:57.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.068+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe8440fae60 con 0x7fe8440f5410 2024-01-27T23:27:57.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.069+0000 7fe8417fa700 1 -- 172.21.15.194:0/599732344 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe83801d7f0 con 0x7fe8440f5410 2024-01-27T23:27:57.059 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:57.059 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:57.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.070+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe83402ff60 msgr2=0x7fe834032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.070+0000 7fe84afe1700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe83402ff60 0x7fe834032420 secure :-1 s=READY pgs=56 cs=0 l=1 rev1=1 rx=0x7fe8300096a0 tx=0x7fe830006b40).stop 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.070+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 msgr2=0x7fe844105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.070+0000 7fe84afe1700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe844105630 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fe838013510 tx=0x7fe83800b040).stop 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.071+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 shutdown_connections 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.071+0000 7fe84afe1700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe83402ff60 0x7fe834032420 unknown :-1 s=CLOSED pgs=56 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.071+0000 7fe84afe1700 1 --2- 172.21.15.194:0/599732344 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe8440f5410 0x7fe844105630 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.071+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 >> 172.21.15.194:0/599732344 conn(0x7fe8440f0b70 msgr2=0x7fe8440f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.071+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 shutdown_connections 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:57.071+0000 7fe84afe1700 1 -- 172.21.15.194:0/599732344 wait complete. 2024-01-27T23:27:57.061 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:27:57.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:57 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:57.070369+0000 mon.smithi121 (mon.0) 167 : audit [DBG] from='client.? 172.21.15.194:0/599732344' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:27:58.432 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:27:58.432 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:27:58.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:27:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:57.280421+0000 mgr.smithi121.hcppvz (mgr.14162) 81 : cluster [DBG] pgmap v74: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:27:59.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 -- 172.21.15.194:0/3313615899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 msgr2=0x7fe2040f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:59.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 --2- 172.21.15.194:0/3313615899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2040f66c0 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7fe1ec009fb0 tx=0x7fe1ec009b30).stop 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 -- 172.21.15.194:0/3313615899 shutdown_connections 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 --2- 172.21.15.194:0/3313615899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2040f66c0 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 -- 172.21.15.194:0/3313615899 >> 172.21.15.194:0/3313615899 conn(0x7fe2040f1aa0 msgr2=0x7fe2040f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 -- 172.21.15.194:0/3313615899 shutdown_connections 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 -- 172.21.15.194:0/3313615899 wait complete. 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 Processor -- start 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 -- start start 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.569+0000 7fe208e1a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2041064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:59.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe208e1a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe2041069a0 con 0x7fe2040f62a0 2024-01-27T23:27:59.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe2037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2041064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:59.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe2037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2041064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:36268/0 (socket says 172.21.15.194:36268) 2024-01-27T23:27:59.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe2037fe700 1 -- 172.21.15.194:0/1344111921 learned_addr learned my addr 172.21.15.194:0/1344111921 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:27:59.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe2037fe700 1 -- 172.21.15.194:0/1344111921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe1ec005040 con 0x7fe2040f62a0 2024-01-27T23:27:59.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe2037fe700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2041064b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe1ec013510 tx=0x7fe1ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:59.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe1fbfff700 1 -- 172.21.15.194:0/1344111921 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1ec009320 con 0x7fe2040f62a0 2024-01-27T23:27:59.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe204106ba0 con 0x7fe2040f62a0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe1fbfff700 1 -- 172.21.15.194:0/1344111921 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe1ec013f00 con 0x7fe2040f62a0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe1fbfff700 1 -- 172.21.15.194:0/1344111921 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe1ec014dc0 con 0x7fe2040f62a0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.570+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe204107030 con 0x7fe2040f62a0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.571+0000 7fe1fbfff700 1 -- 172.21.15.194:0/1344111921 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe1ec01d3d0 con 0x7fe2040f62a0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.571+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe204045ad0 con 0x7fe2040f62a0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.571+0000 7fe1fbfff700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe1f002ff60 0x7fe1f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.571+0000 7fe1fbfff700 1 -- 172.21.15.194:0/1344111921 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe1ec041300 con 0x7fe2040f62a0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.571+0000 7fe202ffd700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe1f002ff60 0x7fe1f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:27:59.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.572+0000 7fe202ffd700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe1f002ff60 0x7fe1f0032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe1f40096a0 tx=0x7fe1f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:27:59.565 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.576+0000 7fe1fbfff700 1 -- 172.21.15.194:0/1344111921 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe1ec01ccb0 con 0x7fe2040f62a0 2024-01-27T23:27:59.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.781+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe204107ee0 con 0x7fe2040f62a0 2024-01-27T23:27:59.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.781+0000 7fe1fbfff700 1 -- 172.21.15.194:0/1344111921 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe1ec01cad0 con 0x7fe2040f62a0 2024-01-27T23:27:59.772 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:27:59.772 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:27:59.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe1f002ff60 msgr2=0x7fe1f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:59.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe1f002ff60 0x7fe1f0032420 secure :-1 s=READY pgs=57 cs=0 l=1 rev1=1 rx=0x7fe1f40096a0 tx=0x7fe1f4006b40).stop 2024-01-27T23:27:59.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 msgr2=0x7fe2041064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:27:59.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2041064b0 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7fe1ec013510 tx=0x7fe1ec00b040).stop 2024-01-27T23:27:59.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 shutdown_connections 2024-01-27T23:27:59.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe1f002ff60 0x7fe1f0032420 unknown :-1 s=CLOSED pgs=57 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:59.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 --2- 172.21.15.194:0/1344111921 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2040f62a0 0x7fe2041064b0 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:27:59.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 >> 172.21.15.194:0/1344111921 conn(0x7fe2040f1aa0 msgr2=0x7fe2040f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:27:59.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 shutdown_connections 2024-01-27T23:27:59.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:27:59.783+0000 7fe208e1a700 1 -- 172.21.15.194:0/1344111921 wait complete. 2024-01-27T23:27:59.774 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:00.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:27:59.280679+0000 mgr.smithi121.hcppvz (mgr.14162) 82 : cluster [DBG] pgmap v75: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:00.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:27:59.782878+0000 mon.smithi121 (mon.0) 168 : audit [DBG] from='client.? 172.21.15.194:0/1344111921' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:01.147 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:01.148 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:02.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.368+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/1627550927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 msgr2=0x7f8cc80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:02.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.368+0000 7f8ccd7e2700 1 --2- 172.21.15.194:0/1627550927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc80f5830 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f8cbc009fb0 tx=0x7f8cbc009b30).stop 2024-01-27T23:28:02.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.368+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/1627550927 shutdown_connections 2024-01-27T23:28:02.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.368+0000 7f8ccd7e2700 1 --2- 172.21.15.194:0/1627550927 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc80f5830 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:02.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.368+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/1627550927 >> 172.21.15.194:0/1627550927 conn(0x7f8cc80f0b70 msgr2=0x7f8cc80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:02.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.368+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/1627550927 shutdown_connections 2024-01-27T23:28:02.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.368+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/1627550927 wait complete. 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8ccd7e2700 1 Processor -- start 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8ccd7e2700 1 -- start start 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8ccd7e2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc8105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8ccd7e2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cc8105b20 con 0x7f8cc80f5410 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8cc6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc8105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8cc6ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc8105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:36288/0 (socket says 172.21.15.194:36288) 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8cc6ffd700 1 -- 172.21.15.194:0/538116833 learned_addr learned my addr 172.21.15.194:0/538116833 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8cc6ffd700 1 -- 172.21.15.194:0/538116833 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8cbc005040 con 0x7f8cc80f5410 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.369+0000 7f8cc6ffd700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc8105630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f8cbc013510 tx=0x7f8cbc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.370+0000 7f8caf7fe700 1 -- 172.21.15.194:0/538116833 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cbc009320 con 0x7f8cc80f5410 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.370+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cc8105d20 con 0x7f8cc80f5410 2024-01-27T23:28:02.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.370+0000 7f8caf7fe700 1 -- 172.21.15.194:0/538116833 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8cbc013f00 con 0x7f8cc80f5410 2024-01-27T23:28:02.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.370+0000 7f8caf7fe700 1 -- 172.21.15.194:0/538116833 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8cbc014d90 con 0x7f8cc80f5410 2024-01-27T23:28:02.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.370+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cc81061b0 con 0x7f8cc80f5410 2024-01-27T23:28:02.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.370+0000 7f8caf7fe700 1 -- 172.21.15.194:0/538116833 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8cbc0263a0 con 0x7f8cc80f5410 2024-01-27T23:28:02.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.370+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cc8045ad0 con 0x7f8cc80f5410 2024-01-27T23:28:02.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.371+0000 7f8caf7fe700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cb002ff60 0x7f8cb0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:02.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.371+0000 7f8caf7fe700 1 -- 172.21.15.194:0/538116833 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8cbc041410 con 0x7f8cc80f5410 2024-01-27T23:28:02.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.371+0000 7f8cc67fc700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cb002ff60 0x7f8cb0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:02.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.371+0000 7f8cc67fc700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cb002ff60 0x7f8cb0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8cb80096a0 tx=0x7f8cb8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:02.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.375+0000 7f8caf7fe700 1 -- 172.21.15.194:0/538116833 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8cbc012330 con 0x7f8cc80f5410 2024-01-27T23:28:02.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.579+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cc80fae60 con 0x7f8cc80f5410 2024-01-27T23:28:02.569 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.579+0000 7f8caf7fe700 1 -- 172.21.15.194:0/538116833 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8cbc01ccb0 con 0x7f8cc80f5410 2024-01-27T23:28:02.570 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:02.570 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:02.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cb002ff60 msgr2=0x7f8cb0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:02.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cb002ff60 0x7f8cb0032420 secure :-1 s=READY pgs=58 cs=0 l=1 rev1=1 rx=0x7f8cb80096a0 tx=0x7f8cb8006b40).stop 2024-01-27T23:28:02.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 msgr2=0x7f8cc8105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:02.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc8105630 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f8cbc013510 tx=0x7f8cbc00b040).stop 2024-01-27T23:28:02.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 shutdown_connections 2024-01-27T23:28:02.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cb002ff60 0x7f8cb0032420 unknown :-1 s=CLOSED pgs=58 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:02.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 --2- 172.21.15.194:0/538116833 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cc80f5410 0x7f8cc8105630 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:02.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 >> 172.21.15.194:0/538116833 conn(0x7f8cc80f0b70 msgr2=0x7f8cc80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:02.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 shutdown_connections 2024-01-27T23:28:02.572 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:02.581+0000 7f8ccd7e2700 1 -- 172.21.15.194:0/538116833 wait complete. 2024-01-27T23:28:02.572 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:02.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:01.281073+0000 mgr.smithi121.hcppvz (mgr.14162) 83 : cluster [DBG] pgmap v76: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:03.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:03 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:02.580879+0000 mon.smithi121 (mon.0) 169 : audit [DBG] from='client.? 172.21.15.194:0/538116833' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:03.965 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:03.965 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:04.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:03.281379+0000 mgr.smithi121.hcppvz (mgr.14162) 84 : cluster [DBG] pgmap v77: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 -- 172.21.15.194:0/737197986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 msgr2=0x7f02e80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 --2- 172.21.15.194:0/737197986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e80f6930 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f02dc009fb0 tx=0x7f02dc009b30).stop 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 -- 172.21.15.194:0/737197986 shutdown_connections 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 --2- 172.21.15.194:0/737197986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e80f6930 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 -- 172.21.15.194:0/737197986 >> 172.21.15.194:0/737197986 conn(0x7f02e80f1c70 msgr2=0x7f02e80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 -- 172.21.15.194:0/737197986 shutdown_connections 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 -- 172.21.15.194:0/737197986 wait complete. 2024-01-27T23:28:05.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.125+0000 7f02eeb74700 1 Processor -- start 2024-01-27T23:28:05.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.126+0000 7f02eeb74700 1 -- start start 2024-01-27T23:28:05.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.126+0000 7f02eeb74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:05.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.126+0000 7f02eeb74700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02e8106c20 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.126+0000 7f02ec910700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.126+0000 7f02ec910700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:36310/0 (socket says 172.21.15.194:36310) 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.126+0000 7f02ec910700 1 -- 172.21.15.194:0/380118297 learned_addr learned my addr 172.21.15.194:0/380118297 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.127+0000 7f02ec910700 1 -- 172.21.15.194:0/380118297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f02dc005040 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.127+0000 7f02ec910700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e8106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f02dc014000 tx=0x7f02dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.127+0000 7f02e57fa700 1 -- 172.21.15.194:0/380118297 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02dc0095f0 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.127+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f02e8106e20 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.127+0000 7f02e57fa700 1 -- 172.21.15.194:0/380118297 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f02dc007e70 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.127+0000 7f02e57fa700 1 -- 172.21.15.194:0/380118297 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f02dc014ea0 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.127+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f02e81072b0 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.128+0000 7f02e57fa700 1 -- 172.21.15.194:0/380118297 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f02dc025070 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.128+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f02e8045ad0 con 0x7f02e80f6510 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.128+0000 7f02e57fa700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:05.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.128+0000 7f02e57fa700 1 -- 172.21.15.194:0/380118297 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f02dc0409f0 con 0x7f02e80f6510 2024-01-27T23:28:05.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.128+0000 7f02e7fff700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:05.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.128+0000 7f02e7fff700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f02d802ff60 0x7f02d8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f02d40096a0 tx=0x7f02d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:05.122 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.132+0000 7f02e57fa700 1 -- 172.21.15.194:0/380118297 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f02dc00f4e0 con 0x7f02e80f6510 2024-01-27T23:28:05.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.338+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f02e80fbf60 con 0x7f02e80f6510 2024-01-27T23:28:05.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.338+0000 7f02e57fa700 1 -- 172.21.15.194:0/380118297 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f02dc00f300 con 0x7f02e80f6510 2024-01-27T23:28:05.329 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:05.329 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:05.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f02d802ff60 msgr2=0x7f02d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:05.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f02d802ff60 0x7f02d8032420 secure :-1 s=READY pgs=59 cs=0 l=1 rev1=1 rx=0x7f02d40096a0 tx=0x7f02d4006b40).stop 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 msgr2=0x7f02e8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e8106730 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f02dc014000 tx=0x7f02dc00b040).stop 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 shutdown_connections 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f02d802ff60 0x7f02d8032420 unknown :-1 s=CLOSED pgs=59 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 --2- 172.21.15.194:0/380118297 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02e80f6510 0x7f02e8106730 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 >> 172.21.15.194:0/380118297 conn(0x7f02e80f1c70 msgr2=0x7f02e80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 shutdown_connections 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:05.340+0000 7f02eeb74700 1 -- 172.21.15.194:0/380118297 wait complete. 2024-01-27T23:28:05.331 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:06.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:05.281672+0000 mgr.smithi121.hcppvz (mgr.14162) 85 : cluster [DBG] pgmap v78: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:06.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:05.339711+0000 mon.smithi121 (mon.0) 170 : audit [DBG] from='client.? 172.21.15.194:0/380118297' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:06.732 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:06.733 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:07.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 -- 172.21.15.194:0/981718786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 msgr2=0x7f7f680f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:07.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 --2- 172.21.15.194:0/981718786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f680f55c0 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f7f5c009fb0 tx=0x7f7f5c009b30).stop 2024-01-27T23:28:07.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 -- 172.21.15.194:0/981718786 shutdown_connections 2024-01-27T23:28:07.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 --2- 172.21.15.194:0/981718786 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f680f55c0 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:07.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 -- 172.21.15.194:0/981718786 >> 172.21.15.194:0/981718786 conn(0x7f7f680f09a0 msgr2=0x7f7f680f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:07.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 -- 172.21.15.194:0/981718786 shutdown_connections 2024-01-27T23:28:07.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 -- 172.21.15.194:0/981718786 wait complete. 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 Processor -- start 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.843+0000 7f7f6759e700 1 -- start start 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.844+0000 7f7f6759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f681091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.844+0000 7f7f6759e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7f68060240 con 0x7f7f680f51a0 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.844+0000 7f7f6659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f681091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.844+0000 7f7f6659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f681091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:39642/0 (socket says 172.21.15.194:39642) 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.844+0000 7f7f6659c700 1 -- 172.21.15.194:0/2290967649 learned_addr learned my addr 172.21.15.194:0/2290967649 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.844+0000 7f7f6659c700 1 -- 172.21.15.194:0/2290967649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7f5c005040 con 0x7f7f680f51a0 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.844+0000 7f7f6659c700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f681091b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7f5c013510 tx=0x7f7f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:07.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.845+0000 7f7f56ffd700 1 -- 172.21.15.194:0/2290967649 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f5c009320 con 0x7f7f680f51a0 2024-01-27T23:28:07.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.845+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7f68060440 con 0x7f7f680f51a0 2024-01-27T23:28:07.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.845+0000 7f7f56ffd700 1 -- 172.21.15.194:0/2290967649 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7f5c013f00 con 0x7f7f680f51a0 2024-01-27T23:28:07.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.845+0000 7f7f56ffd700 1 -- 172.21.15.194:0/2290967649 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7f5c014dc0 con 0x7f7f680f51a0 2024-01-27T23:28:07.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.845+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7f68109860 con 0x7f7f680f51a0 2024-01-27T23:28:07.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.845+0000 7f7f56ffd700 1 -- 172.21.15.194:0/2290967649 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7f5c0095f0 con 0x7f7f680f51a0 2024-01-27T23:28:07.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.845+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7f68045ad0 con 0x7f7f680f51a0 2024-01-27T23:28:07.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.846+0000 7f7f56ffd700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7f5002ff60 0x7f7f50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:07.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.846+0000 7f7f56ffd700 1 -- 172.21.15.194:0/2290967649 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7f5c041000 con 0x7f7f680f51a0 2024-01-27T23:28:07.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.846+0000 7f7f65d9b700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7f5002ff60 0x7f7f50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:07.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.846+0000 7f7f65d9b700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7f5002ff60 0x7f7f50032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7f580096a0 tx=0x7f7f58006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:07.840 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:07.850+0000 7f7f56ffd700 1 -- 172.21.15.194:0/2290967649 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7f5c014420 con 0x7f7f680f51a0 2024-01-27T23:28:08.050 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.060+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7f6810a080 con 0x7f7f680f51a0 2024-01-27T23:28:08.051 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.061+0000 7f7f56ffd700 1 -- 172.21.15.194:0/2290967649 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7f5c010230 con 0x7f7f680f51a0 2024-01-27T23:28:08.051 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:08.051 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7f5002ff60 msgr2=0x7f7f50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7f5002ff60 0x7f7f50032420 secure :-1 s=READY pgs=60 cs=0 l=1 rev1=1 rx=0x7f7f580096a0 tx=0x7f7f58006b40).stop 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 msgr2=0x7f7f681091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f681091b0 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f7f5c013510 tx=0x7f7f5c00b040).stop 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 shutdown_connections 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7f5002ff60 0x7f7f50032420 unknown :-1 s=CLOSED pgs=60 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 --2- 172.21.15.194:0/2290967649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7f680f51a0 0x7f7f681091b0 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 >> 172.21.15.194:0/2290967649 conn(0x7f7f680f09a0 msgr2=0x7f7f680f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 shutdown_connections 2024-01-27T23:28:08.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:08.062+0000 7f7f6759e700 1 -- 172.21.15.194:0/2290967649 wait complete. 2024-01-27T23:28:08.054 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:08.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:07.282081+0000 mgr.smithi121.hcppvz (mgr.14162) 86 : cluster [DBG] pgmap v79: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:08.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:08.062061+0000 mon.smithi121 (mon.0) 171 : audit [DBG] from='client.? 172.21.15.194:0/2290967649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:09.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:09.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:10.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:09.282454+0000 mgr.smithi121.hcppvz (mgr.14162) 87 : cluster [DBG] pgmap v80: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 -- 172.21.15.194:0/3470355480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 msgr2=0x7f04140fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 --2- 172.21.15.194:0/3470355480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04140fb550 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f0408009fb0 tx=0x7f0408009b30).stop 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 -- 172.21.15.194:0/3470355480 shutdown_connections 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 --2- 172.21.15.194:0/3470355480 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04140fb550 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 -- 172.21.15.194:0/3470355480 >> 172.21.15.194:0/3470355480 conn(0x7f04140f0b70 msgr2=0x7f04140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 -- 172.21.15.194:0/3470355480 shutdown_connections 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 -- 172.21.15.194:0/3470355480 wait complete. 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.627+0000 7f041add8700 1 Processor -- start 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.628+0000 7f041add8700 1 -- start start 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.628+0000 7f041add8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04141088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.628+0000 7f041add8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0414108da0 con 0x7f0414060220 2024-01-27T23:28:10.619 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.628+0000 7f0418b74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04141088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:10.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.628+0000 7f0418b74700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04141088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:39654/0 (socket says 172.21.15.194:39654) 2024-01-27T23:28:10.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.628+0000 7f0418b74700 1 -- 172.21.15.194:0/3555469651 learned_addr learned my addr 172.21.15.194:0/3555469651 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:10.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.629+0000 7f0418b74700 1 -- 172.21.15.194:0/3555469651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0408005040 con 0x7f0414060220 2024-01-27T23:28:10.620 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.629+0000 7f0418b74700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04141088b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f0408013510 tx=0x7f040800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:10.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.629+0000 7f04117fa700 1 -- 172.21.15.194:0/3555469651 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0408009320 con 0x7f0414060220 2024-01-27T23:28:10.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.629+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0414108fa0 con 0x7f0414060220 2024-01-27T23:28:10.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.629+0000 7f04117fa700 1 -- 172.21.15.194:0/3555469651 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0408013f00 con 0x7f0414060220 2024-01-27T23:28:10.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.629+0000 7f04117fa700 1 -- 172.21.15.194:0/3555469651 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0408014dc0 con 0x7f0414060220 2024-01-27T23:28:10.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.629+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0414109430 con 0x7f0414060220 2024-01-27T23:28:10.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.630+0000 7f04117fa700 1 -- 172.21.15.194:0/3555469651 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0408014420 con 0x7f0414060220 2024-01-27T23:28:10.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.630+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0414045ad0 con 0x7f0414060220 2024-01-27T23:28:10.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.630+0000 7f04117fa700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f040402ff60 0x7f0404032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:10.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.630+0000 7f04117fa700 1 -- 172.21.15.194:0/3555469651 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0408041020 con 0x7f0414060220 2024-01-27T23:28:10.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.630+0000 7f0413fff700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f040402ff60 0x7f0404032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:10.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.631+0000 7f0413fff700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f040402ff60 0x7f0404032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f04000096a0 tx=0x7f0400006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:10.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.634+0000 7f04117fa700 1 -- 172.21.15.194:0/3555469651 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f04080095f0 con 0x7f0414060220 2024-01-27T23:28:10.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.843+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f041410a2e0 con 0x7f0414060220 2024-01-27T23:28:10.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.843+0000 7f04117fa700 1 -- 172.21.15.194:0/3555469651 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f040801cb10 con 0x7f0414060220 2024-01-27T23:28:10.834 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:10.834 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:10.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f040402ff60 msgr2=0x7f0404032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:10.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f040402ff60 0x7f0404032420 secure :-1 s=READY pgs=61 cs=0 l=1 rev1=1 rx=0x7f04000096a0 tx=0x7f0400006b40).stop 2024-01-27T23:28:10.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 msgr2=0x7f04141088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:10.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04141088b0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f0408013510 tx=0x7f040800b040).stop 2024-01-27T23:28:10.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 shutdown_connections 2024-01-27T23:28:10.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f040402ff60 0x7f0404032420 unknown :-1 s=CLOSED pgs=61 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:10.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 --2- 172.21.15.194:0/3555469651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0414060220 0x7f04141088b0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:10.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 >> 172.21.15.194:0/3555469651 conn(0x7f04140f0b70 msgr2=0x7f04140f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:10.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 shutdown_connections 2024-01-27T23:28:10.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:10.845+0000 7f041add8700 1 -- 172.21.15.194:0/3555469651 wait complete. 2024-01-27T23:28:10.836 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:11.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:11 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:10.844770+0000 mon.smithi121 (mon.0) 172 : audit [DBG] from='client.? 172.21.15.194:0/3555469651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:12.220 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:12.220 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:12.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:11.282858+0000 mgr.smithi121.hcppvz (mgr.14162) 88 : cluster [DBG] pgmap v81: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:13.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.358+0000 7fcb529d5700 1 -- 172.21.15.194:0/2849262026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 msgr2=0x7fcb4c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:13.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.358+0000 7fcb529d5700 1 --2- 172.21.15.194:0/2849262026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c0f6930 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fcb3c009fb0 tx=0x7fcb3c009b30).stop 2024-01-27T23:28:13.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.358+0000 7fcb529d5700 1 -- 172.21.15.194:0/2849262026 shutdown_connections 2024-01-27T23:28:13.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.358+0000 7fcb529d5700 1 --2- 172.21.15.194:0/2849262026 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c0f6930 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:13.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.358+0000 7fcb529d5700 1 -- 172.21.15.194:0/2849262026 >> 172.21.15.194:0/2849262026 conn(0x7fcb4c0f1c70 msgr2=0x7fcb4c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:13.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.358+0000 7fcb529d5700 1 -- 172.21.15.194:0/2849262026 shutdown_connections 2024-01-27T23:28:13.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.358+0000 7fcb529d5700 1 -- 172.21.15.194:0/2849262026 wait complete. 2024-01-27T23:28:13.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.359+0000 7fcb529d5700 1 Processor -- start 2024-01-27T23:28:13.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.359+0000 7fcb529d5700 1 -- start start 2024-01-27T23:28:13.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.359+0000 7fcb529d5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:13.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.359+0000 7fcb529d5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcb4c106c20 con 0x7fcb4c0f6510 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.359+0000 7fcb4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.359+0000 7fcb4bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:39678/0 (socket says 172.21.15.194:39678) 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.359+0000 7fcb4bfff700 1 -- 172.21.15.194:0/3474091510 learned_addr learned my addr 172.21.15.194:0/3474091510 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.360+0000 7fcb4bfff700 1 -- 172.21.15.194:0/3474091510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcb3c005040 con 0x7fcb4c0f6510 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.360+0000 7fcb4bfff700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcb3c014000 tx=0x7fcb3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.360+0000 7fcb48ff9700 1 -- 172.21.15.194:0/3474091510 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c0095f0 con 0x7fcb4c0f6510 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.360+0000 7fcb48ff9700 1 -- 172.21.15.194:0/3474091510 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcb3c007e70 con 0x7fcb4c0f6510 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.360+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcb4c106e20 con 0x7fcb4c0f6510 2024-01-27T23:28:13.352 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.360+0000 7fcb48ff9700 1 -- 172.21.15.194:0/3474091510 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcb3c014ea0 con 0x7fcb4c0f6510 2024-01-27T23:28:13.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.360+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcb4c1072b0 con 0x7fcb4c0f6510 2024-01-27T23:28:13.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.361+0000 7fcb48ff9700 1 -- 172.21.15.194:0/3474091510 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcb3c0079c0 con 0x7fcb4c0f6510 2024-01-27T23:28:13.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.361+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcb4c045ad0 con 0x7fcb4c0f6510 2024-01-27T23:28:13.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.361+0000 7fcb48ff9700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcb3402ff60 0x7fcb34032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:13.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.361+0000 7fcb48ff9700 1 -- 172.21.15.194:0/3474091510 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcb3c040860 con 0x7fcb4c0f6510 2024-01-27T23:28:13.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.361+0000 7fcb4b7fe700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcb3402ff60 0x7fcb34032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:13.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.361+0000 7fcb4b7fe700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcb3402ff60 0x7fcb34032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcb400096a0 tx=0x7fcb40006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:13.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.365+0000 7fcb48ff9700 1 -- 172.21.15.194:0/3474091510 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcb3c00f950 con 0x7fcb4c0f6510 2024-01-27T23:28:13.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.572+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcb4c0fbf60 con 0x7fcb4c0f6510 2024-01-27T23:28:13.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.573+0000 7fcb48ff9700 1 -- 172.21.15.194:0/3474091510 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcb3c010230 con 0x7fcb4c0f6510 2024-01-27T23:28:13.564 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:13.565 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:13.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.574+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcb3402ff60 msgr2=0x7fcb34032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:13.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcb3402ff60 0x7fcb34032420 secure :-1 s=READY pgs=62 cs=0 l=1 rev1=1 rx=0x7fcb400096a0 tx=0x7fcb40006b40).stop 2024-01-27T23:28:13.566 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 msgr2=0x7fcb4c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:13.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c106730 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fcb3c014000 tx=0x7fcb3c00b040).stop 2024-01-27T23:28:13.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 shutdown_connections 2024-01-27T23:28:13.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcb3402ff60 0x7fcb34032420 unknown :-1 s=CLOSED pgs=62 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:13.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 --2- 172.21.15.194:0/3474091510 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcb4c0f6510 0x7fcb4c106730 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:13.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 >> 172.21.15.194:0/3474091510 conn(0x7fcb4c0f1c70 msgr2=0x7fcb4c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:13.567 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 shutdown_connections 2024-01-27T23:28:13.568 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:13.575+0000 7fcb529d5700 1 -- 172.21.15.194:0/3474091510 wait complete. 2024-01-27T23:28:13.568 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:14.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:13.283172+0000 mgr.smithi121.hcppvz (mgr.14162) 89 : cluster [DBG] pgmap v82: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:14.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:13.574399+0000 mon.smithi121 (mon.0) 173 : audit [DBG] from='client.? 172.21.15.194:0/3474091510' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:14.938 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:14.938 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:16.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.102+0000 7f8f68896700 1 -- 172.21.15.194:0/2378001734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 msgr2=0x7f8f640f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:16.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.102+0000 7f8f68896700 1 --2- 172.21.15.194:0/2378001734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f640f6930 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f8f54009fb0 tx=0x7f8f54009b30).stop 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.102+0000 7f8f68896700 1 -- 172.21.15.194:0/2378001734 shutdown_connections 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.102+0000 7f8f68896700 1 --2- 172.21.15.194:0/2378001734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f640f6930 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.102+0000 7f8f68896700 1 -- 172.21.15.194:0/2378001734 >> 172.21.15.194:0/2378001734 conn(0x7f8f640f1c70 msgr2=0x7f8f640f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f68896700 1 -- 172.21.15.194:0/2378001734 shutdown_connections 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f68896700 1 -- 172.21.15.194:0/2378001734 wait complete. 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f68896700 1 Processor -- start 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f68896700 1 -- start start 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f68896700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f6406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f68896700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8f64069e20 con 0x7f8f640f6510 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f6259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f6406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f6259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f6406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:56602/0 (socket says 172.21.15.194:56602) 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.103+0000 7f8f6259c700 1 -- 172.21.15.194:0/3910445733 learned_addr learned my addr 172.21.15.194:0/3910445733 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.104+0000 7f8f6259c700 1 -- 172.21.15.194:0/3910445733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8f54005040 con 0x7f8f640f6510 2024-01-27T23:28:16.095 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.104+0000 7f8f6259c700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f6406b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8f54013510 tx=0x7f8f5400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:16.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.104+0000 7f8f5affd700 1 -- 172.21.15.194:0/3910445733 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f54009320 con 0x7f8f640f6510 2024-01-27T23:28:16.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.104+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8f64069fc0 con 0x7f8f640f6510 2024-01-27T23:28:16.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.104+0000 7f8f5affd700 1 -- 172.21.15.194:0/3910445733 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8f54013f00 con 0x7f8f640f6510 2024-01-27T23:28:16.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.104+0000 7f8f5affd700 1 -- 172.21.15.194:0/3910445733 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8f54014d90 con 0x7f8f640f6510 2024-01-27T23:28:16.096 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.104+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8f6406a450 con 0x7f8f640f6510 2024-01-27T23:28:16.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.105+0000 7f8f5affd700 1 -- 172.21.15.194:0/3910445733 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8f54014420 con 0x7f8f640f6510 2024-01-27T23:28:16.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.105+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8f64059100 con 0x7f8f640f6510 2024-01-27T23:28:16.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.105+0000 7f8f5affd700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8f5002ff60 0x7f8f50032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:16.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.105+0000 7f8f5affd700 1 -- 172.21.15.194:0/3910445733 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8f54040ff0 con 0x7f8f640f6510 2024-01-27T23:28:16.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.105+0000 7f8f61d9b700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8f5002ff60 0x7f8f50032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:16.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.105+0000 7f8f61d9b700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8f5002ff60 0x7f8f50032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8f4c0096a0 tx=0x7f8f4c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:16.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.109+0000 7f8f5affd700 1 -- 172.21.15.194:0/3910445733 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8f5406d2b0 con 0x7f8f640f6510 2024-01-27T23:28:16.307 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.317+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8f640fbf60 con 0x7f8f640f6510 2024-01-27T23:28:16.308 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.318+0000 7f8f5affd700 1 -- 172.21.15.194:0/3910445733 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8f5401db70 con 0x7f8f640f6510 2024-01-27T23:28:16.308 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:16.308 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8f5002ff60 msgr2=0x7f8f50032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8f5002ff60 0x7f8f50032420 secure :-1 s=READY pgs=63 cs=0 l=1 rev1=1 rx=0x7f8f4c0096a0 tx=0x7f8f4c006b40).stop 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 msgr2=0x7f8f6406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f6406b9a0 secure :-1 s=READY pgs=194 cs=0 l=1 rev1=1 rx=0x7f8f54013510 tx=0x7f8f5400b040).stop 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 shutdown_connections 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8f5002ff60 0x7f8f50032420 unknown :-1 s=CLOSED pgs=63 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 --2- 172.21.15.194:0/3910445733 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8f640f6510 0x7f8f6406b9a0 unknown :-1 s=CLOSED pgs=194 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 >> 172.21.15.194:0/3910445733 conn(0x7f8f640f1c70 msgr2=0x7f8f640f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 shutdown_connections 2024-01-27T23:28:16.310 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:16.319+0000 7f8f68896700 1 -- 172.21.15.194:0/3910445733 wait complete. 2024-01-27T23:28:16.311 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:16.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:15.283521+0000 mgr.smithi121.hcppvz (mgr.14162) 90 : cluster [DBG] pgmap v83: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:17.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:16.319024+0000 mon.smithi121 (mon.0) 174 : audit [DBG] from='client.? 172.21.15.194:0/3910445733' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:17.697 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:17.697 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:18.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:17.283954+0000 mgr.smithi121.hcppvz (mgr.14162) 91 : cluster [DBG] pgmap v84: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:18.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:17.394104+0000 mon.smithi121 (mon.0) 175 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:28:18.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:17.402672+0000 mon.smithi121 (mon.0) 176 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:28:18.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.852+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/362587540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 msgr2=0x7ff2f00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:18.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.852+0000 7ff2f4cf5700 1 --2- 172.21.15.194:0/362587540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f00f5830 secure :-1 s=READY pgs=195 cs=0 l=1 rev1=1 rx=0x7ff2e0009fb0 tx=0x7ff2e0009b30).stop 2024-01-27T23:28:18.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.852+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/362587540 shutdown_connections 2024-01-27T23:28:18.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.852+0000 7ff2f4cf5700 1 --2- 172.21.15.194:0/362587540 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f00f5830 unknown :-1 s=CLOSED pgs=195 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:18.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.852+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/362587540 >> 172.21.15.194:0/362587540 conn(0x7ff2f00f0b70 msgr2=0x7ff2f00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:18.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.852+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/362587540 shutdown_connections 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.852+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/362587540 wait complete. 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2f4cf5700 1 Processor -- start 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2f4cf5700 1 -- start start 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2f4cf5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2f4cf5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff2f0060240 con 0x7ff2f00f5410 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2ee59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2ee59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:56622/0 (socket says 172.21.15.194:56622) 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2ee59c700 1 -- 172.21.15.194:0/2178334061 learned_addr learned my addr 172.21.15.194:0/2178334061 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2ee59c700 1 -- 172.21.15.194:0/2178334061 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff2e0005040 con 0x7ff2f00f5410 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.853+0000 7ff2ee59c700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f0109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff2e0013510 tx=0x7ff2e000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.854+0000 7ff2deffd700 1 -- 172.21.15.194:0/2178334061 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e0009320 con 0x7ff2f00f5410 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.854+0000 7ff2deffd700 1 -- 172.21.15.194:0/2178334061 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff2e0013f00 con 0x7ff2f00f5410 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.854+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff2f0060440 con 0x7ff2f00f5410 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.854+0000 7ff2deffd700 1 -- 172.21.15.194:0/2178334061 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff2e0014d90 con 0x7ff2f00f5410 2024-01-27T23:28:18.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.854+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff2f0109ae0 con 0x7ff2f00f5410 2024-01-27T23:28:18.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.855+0000 7ff2deffd700 1 -- 172.21.15.194:0/2178334061 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff2e00095f0 con 0x7ff2f00f5410 2024-01-27T23:28:18.846 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.855+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff2f0045ad0 con 0x7ff2f00f5410 2024-01-27T23:28:18.847 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.855+0000 7ff2deffd700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff2d802ff60 0x7ff2d8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:18.847 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.855+0000 7ff2deffd700 1 -- 172.21.15.194:0/2178334061 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff2e0041000 con 0x7ff2f00f5410 2024-01-27T23:28:18.847 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.855+0000 7ff2edd9b700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff2d802ff60 0x7ff2d8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:18.847 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.855+0000 7ff2edd9b700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff2d802ff60 0x7ff2d8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff2e40096a0 tx=0x7ff2e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:18.853 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:18.859+0000 7ff2deffd700 1 -- 172.21.15.194:0/2178334061 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff2e0014420 con 0x7ff2f00f5410 2024-01-27T23:28:19.057 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.067+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff2f00fae60 con 0x7ff2f00f5410 2024-01-27T23:28:19.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.067+0000 7ff2deffd700 1 -- 172.21.15.194:0/2178334061 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff2e0010230 con 0x7ff2f00f5410 2024-01-27T23:28:19.058 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:19.058 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:19.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff2d802ff60 msgr2=0x7ff2d8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:19.059 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff2d802ff60 0x7ff2d8032420 secure :-1 s=READY pgs=64 cs=0 l=1 rev1=1 rx=0x7ff2e40096a0 tx=0x7ff2e4006b40).stop 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 msgr2=0x7ff2f0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f0109430 secure :-1 s=READY pgs=196 cs=0 l=1 rev1=1 rx=0x7ff2e0013510 tx=0x7ff2e000b040).stop 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 shutdown_connections 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff2d802ff60 0x7ff2d8032420 unknown :-1 s=CLOSED pgs=64 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 --2- 172.21.15.194:0/2178334061 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff2f00f5410 0x7ff2f0109430 unknown :-1 s=CLOSED pgs=196 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 >> 172.21.15.194:0/2178334061 conn(0x7ff2f00f0b70 msgr2=0x7ff2f00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 shutdown_connections 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:19.069+0000 7ff2f4cf5700 1 -- 172.21.15.194:0/2178334061 wait complete. 2024-01-27T23:28:19.060 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:19.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:19 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:19.068733+0000 mon.smithi121 (mon.0) 177 : audit [DBG] from='client.? 172.21.15.194:0/2178334061' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:20.455 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:20.455 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:20.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:19.284295+0000 mgr.smithi121.hcppvz (mgr.14162) 92 : cluster [DBG] pgmap v85: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.564+0000 7fd3a8840700 1 -- 172.21.15.194:0/4110851172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 msgr2=0x7fd3a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.564+0000 7fd3a8840700 1 --2- 172.21.15.194:0/4110851172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a40f55c0 secure :-1 s=READY pgs=197 cs=0 l=1 rev1=1 rx=0x7fd394009fb0 tx=0x7fd394009b30).stop 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.564+0000 7fd3a8840700 1 -- 172.21.15.194:0/4110851172 shutdown_connections 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.564+0000 7fd3a8840700 1 --2- 172.21.15.194:0/4110851172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a40f55c0 unknown :-1 s=CLOSED pgs=197 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.564+0000 7fd3a8840700 1 -- 172.21.15.194:0/4110851172 >> 172.21.15.194:0/4110851172 conn(0x7fd3a40f09a0 msgr2=0x7fd3a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.565+0000 7fd3a8840700 1 -- 172.21.15.194:0/4110851172 shutdown_connections 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.565+0000 7fd3a8840700 1 -- 172.21.15.194:0/4110851172 wait complete. 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.565+0000 7fd3a8840700 1 Processor -- start 2024-01-27T23:28:21.556 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.565+0000 7fd3a8840700 1 -- start start 2024-01-27T23:28:21.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.565+0000 7fd3a8840700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a41091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:21.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.565+0000 7fd3a8840700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3a4060220 con 0x7fd3a40f51a0 2024-01-27T23:28:21.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd3a37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a41091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:21.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd3a37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a41091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:56640/0 (socket says 172.21.15.194:56640) 2024-01-27T23:28:21.557 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd3a37fe700 1 -- 172.21.15.194:0/2631107285 learned_addr learned my addr 172.21.15.194:0/2631107285 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd3a37fe700 1 -- 172.21.15.194:0/2631107285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd394005040 con 0x7fd3a40f51a0 2024-01-27T23:28:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd3a37fe700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a41091d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd394009830 tx=0x7fd39400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd38bfff700 1 -- 172.21.15.194:0/2631107285 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd394009320 con 0x7fd3a40f51a0 2024-01-27T23:28:21.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3a4060420 con 0x7fd3a40f51a0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd38bfff700 1 -- 172.21.15.194:0/2631107285 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3940136c0 con 0x7fd3a40f51a0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd38bfff700 1 -- 172.21.15.194:0/2631107285 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd39401ee90 con 0x7fd3a40f51a0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.566+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3a4109880 con 0x7fd3a40f51a0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.567+0000 7fd38bfff700 1 -- 172.21.15.194:0/2631107285 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd39401e420 con 0x7fd3a40f51a0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.567+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3a4059100 con 0x7fd3a40f51a0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.567+0000 7fd38bfff700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd38c02ff10 0x7fd38c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.567+0000 7fd38bfff700 1 -- 172.21.15.194:0/2631107285 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd394041020 con 0x7fd3a40f51a0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.567+0000 7fd3a2ffd700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd38c02ff10 0x7fd38c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:21.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.568+0000 7fd3a2ffd700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd38c02ff10 0x7fd38c0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd3980096a0 tx=0x7fd398006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:21.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.572+0000 7fd38bfff700 1 -- 172.21.15.194:0/2631107285 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd39401c790 con 0x7fd3a40f51a0 2024-01-27T23:28:21.768 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.778+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3a4045ad0 con 0x7fd3a40f51a0 2024-01-27T23:28:21.769 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.779+0000 7fd38bfff700 1 -- 172.21.15.194:0/2631107285 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd39400e2a0 con 0x7fd3a40f51a0 2024-01-27T23:28:21.769 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:21.769 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd38c02ff10 msgr2=0x7fd38c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd38c02ff10 0x7fd38c0323d0 secure :-1 s=READY pgs=65 cs=0 l=1 rev1=1 rx=0x7fd3980096a0 tx=0x7fd398006b40).stop 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 msgr2=0x7fd3a41091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a41091d0 secure :-1 s=READY pgs=198 cs=0 l=1 rev1=1 rx=0x7fd394009830 tx=0x7fd39400b040).stop 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 shutdown_connections 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd38c02ff10 0x7fd38c0323d0 unknown :-1 s=CLOSED pgs=65 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 --2- 172.21.15.194:0/2631107285 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3a40f51a0 0x7fd3a41091d0 unknown :-1 s=CLOSED pgs=198 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 >> 172.21.15.194:0/2631107285 conn(0x7fd3a40f09a0 msgr2=0x7fd3a40f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 shutdown_connections 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:21.781+0000 7fd3a8840700 1 -- 172.21.15.194:0/2631107285 wait complete. 2024-01-27T23:28:21.772 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:22.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:21.284600+0000 mgr.smithi121.hcppvz (mgr.14162) 93 : cluster [DBG] pgmap v86: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:22.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:21.780012+0000 mon.smithi121 (mon.0) 178 : audit [DBG] from='client.? 172.21.15.194:0/2631107285' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:23.149 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:23.149 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:24.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.348+0000 7f1810bbd700 1 -- 172.21.15.194:0/2060949134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 msgr2=0x7f180c0f9170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:24.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.348+0000 7f1810bbd700 1 --2- 172.21.15.194:0/2060949134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c0f9170 secure :-1 s=READY pgs=199 cs=0 l=1 rev1=1 rx=0x7f1800009fb0 tx=0x7f1800009b30).stop 2024-01-27T23:28:24.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.348+0000 7f1810bbd700 1 -- 172.21.15.194:0/2060949134 shutdown_connections 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.348+0000 7f1810bbd700 1 --2- 172.21.15.194:0/2060949134 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c0f9170 unknown :-1 s=CLOSED pgs=199 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.348+0000 7f1810bbd700 1 -- 172.21.15.194:0/2060949134 >> 172.21.15.194:0/2060949134 conn(0x7f180c0f0b70 msgr2=0x7f180c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.348+0000 7f1810bbd700 1 -- 172.21.15.194:0/2060949134 shutdown_connections 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.348+0000 7f1810bbd700 1 -- 172.21.15.194:0/2060949134 wait complete. 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f1810bbd700 1 Processor -- start 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f1810bbd700 1 -- start start 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f1810bbd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c1066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f1810bbd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f180c106b90 con 0x7f180c060200 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f180a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c1066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f180a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c1066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:56654/0 (socket says 172.21.15.194:56654) 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f180a59c700 1 -- 172.21.15.194:0/2484270809 learned_addr learned my addr 172.21.15.194:0/2484270809 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.349+0000 7f180a59c700 1 -- 172.21.15.194:0/2484270809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1800005040 con 0x7f180c060200 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.350+0000 7f180a59c700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c1066a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1800013510 tx=0x7f180000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:24.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.350+0000 7f17faffd700 1 -- 172.21.15.194:0/2484270809 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800009320 con 0x7f180c060200 2024-01-27T23:28:24.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.350+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f180c106d90 con 0x7f180c060200 2024-01-27T23:28:24.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.350+0000 7f17faffd700 1 -- 172.21.15.194:0/2484270809 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1800013f00 con 0x7f180c060200 2024-01-27T23:28:24.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.350+0000 7f17faffd700 1 -- 172.21.15.194:0/2484270809 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1800014dc0 con 0x7f180c060200 2024-01-27T23:28:24.342 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.350+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f180c107220 con 0x7f180c060200 2024-01-27T23:28:24.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.350+0000 7f17faffd700 1 -- 172.21.15.194:0/2484270809 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f180001d3d0 con 0x7f180c060200 2024-01-27T23:28:24.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.351+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f180c045ad0 con 0x7f180c060200 2024-01-27T23:28:24.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.351+0000 7f17faffd700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17f402ff60 0x7f17f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:24.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.351+0000 7f17faffd700 1 -- 172.21.15.194:0/2484270809 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f18000413e0 con 0x7f180c060200 2024-01-27T23:28:24.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.351+0000 7f1809d9b700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17f402ff60 0x7f17f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:24.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.351+0000 7f1809d9b700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17f402ff60 0x7f17f4032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f17fc0096a0 tx=0x7f17fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:24.346 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.355+0000 7f17faffd700 1 -- 172.21.15.194:0/2484270809 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f180001ccb0 con 0x7f180c060200 2024-01-27T23:28:24.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.560+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f180c108070 con 0x7f180c060200 2024-01-27T23:28:24.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.561+0000 7f17faffd700 1 -- 172.21.15.194:0/2484270809 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1800012330 con 0x7f180c060200 2024-01-27T23:28:24.552 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:24.552 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:24.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.562+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17f402ff60 msgr2=0x7f17f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:24.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.562+0000 7f1810bbd700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17f402ff60 0x7f17f4032420 secure :-1 s=READY pgs=66 cs=0 l=1 rev1=1 rx=0x7f17fc0096a0 tx=0x7f17fc006b40).stop 2024-01-27T23:28:24.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.562+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 msgr2=0x7f180c1066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:24.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.562+0000 7f1810bbd700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c1066a0 secure :-1 s=READY pgs=200 cs=0 l=1 rev1=1 rx=0x7f1800013510 tx=0x7f180000b040).stop 2024-01-27T23:28:24.553 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.563+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 shutdown_connections 2024-01-27T23:28:24.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.563+0000 7f1810bbd700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17f402ff60 0x7f17f4032420 unknown :-1 s=CLOSED pgs=66 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:24.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.563+0000 7f1810bbd700 1 --2- 172.21.15.194:0/2484270809 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f180c060200 0x7f180c1066a0 unknown :-1 s=CLOSED pgs=200 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:24.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.563+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 >> 172.21.15.194:0/2484270809 conn(0x7f180c0f0b70 msgr2=0x7f180c0f1620 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:24.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.563+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 shutdown_connections 2024-01-27T23:28:24.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:24.563+0000 7f1810bbd700 1 -- 172.21.15.194:0/2484270809 wait complete. 2024-01-27T23:28:24.554 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:24.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:23.284861+0000 mgr.smithi121.hcppvz (mgr.14162) 94 : cluster [DBG] pgmap v87: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:25.587 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:24.562293+0000 mon.smithi121 (mon.0) 179 : audit [DBG] from='client.? 172.21.15.194:0/2484270809' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:25.891 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:25.891 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:26.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:25.285145+0000 mgr.smithi121.hcppvz (mgr.14162) 95 : cluster [DBG] pgmap v88: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:27.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 -- 172.21.15.194:0/1262327225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 msgr2=0x7f8bb00f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:27.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 --2- 172.21.15.194:0/1262327225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb00f66c0 secure :-1 s=READY pgs=201 cs=0 l=1 rev1=1 rx=0x7f8ba0009fb0 tx=0x7f8ba0009b30).stop 2024-01-27T23:28:27.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 -- 172.21.15.194:0/1262327225 shutdown_connections 2024-01-27T23:28:27.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 --2- 172.21.15.194:0/1262327225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb00f66c0 unknown :-1 s=CLOSED pgs=201 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:27.013 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 -- 172.21.15.194:0/1262327225 >> 172.21.15.194:0/1262327225 conn(0x7f8bb00f1aa0 msgr2=0x7f8bb00f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 -- 172.21.15.194:0/1262327225 shutdown_connections 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 -- 172.21.15.194:0/1262327225 wait complete. 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 Processor -- start 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 -- start start 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb01064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.022+0000 7f8bb6312700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8bb01069a0 con 0x7f8bb00f62a0 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.023+0000 7f8bb5310700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb01064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.023+0000 7f8bb5310700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb01064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48028/0 (socket says 172.21.15.194:48028) 2024-01-27T23:28:27.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.023+0000 7f8bb5310700 1 -- 172.21.15.194:0/2527193828 learned_addr learned my addr 172.21.15.194:0/2527193828 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.023+0000 7f8bb5310700 1 -- 172.21.15.194:0/2527193828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ba0005040 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.023+0000 7f8bb5310700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb01064b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8ba0009830 tx=0x7f8ba000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.023+0000 7f8ba5ffb700 1 -- 172.21.15.194:0/2527193828 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba0009320 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.023+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8bb0106ba0 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.024+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8bb0107030 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.024+0000 7f8ba5ffb700 1 -- 172.21.15.194:0/2527193828 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ba00136c0 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.024+0000 7f8ba5ffb700 1 -- 172.21.15.194:0/2527193828 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ba001ee90 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.024+0000 7f8ba5ffb700 1 -- 172.21.15.194:0/2527193828 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ba001e420 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.024+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8bb0045ad0 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.024+0000 7f8ba5ffb700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8b9c02ff10 0x7f8b9c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.025+0000 7f8ba5ffb700 1 -- 172.21.15.194:0/2527193828 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ba0040fd0 con 0x7f8bb00f62a0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.025+0000 7f8bb4b0f700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8b9c02ff10 0x7f8b9c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:27.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.025+0000 7f8bb4b0f700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8b9c02ff10 0x7f8b9c0323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8bac0096a0 tx=0x7f8bac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:27.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.030+0000 7f8ba5ffb700 1 -- 172.21.15.194:0/2527193828 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ba00095f0 con 0x7f8bb00f62a0 2024-01-27T23:28:27.226 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.236+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8bb0107e30 con 0x7f8bb00f62a0 2024-01-27T23:28:27.227 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.236+0000 7f8ba5ffb700 1 -- 172.21.15.194:0/2527193828 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ba001cb10 con 0x7f8bb00f62a0 2024-01-27T23:28:27.227 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:27.227 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:27.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8b9c02ff10 msgr2=0x7f8b9c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:27.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8b9c02ff10 0x7f8b9c0323d0 secure :-1 s=READY pgs=67 cs=0 l=1 rev1=1 rx=0x7f8bac0096a0 tx=0x7f8bac006b40).stop 2024-01-27T23:28:27.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 msgr2=0x7f8bb01064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:27.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb01064b0 secure :-1 s=READY pgs=202 cs=0 l=1 rev1=1 rx=0x7f8ba0009830 tx=0x7f8ba000b040).stop 2024-01-27T23:28:27.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 shutdown_connections 2024-01-27T23:28:27.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8b9c02ff10 0x7f8b9c0323d0 unknown :-1 s=CLOSED pgs=67 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:27.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 --2- 172.21.15.194:0/2527193828 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8bb00f62a0 0x7f8bb01064b0 unknown :-1 s=CLOSED pgs=202 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:27.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 >> 172.21.15.194:0/2527193828 conn(0x7f8bb00f1aa0 msgr2=0x7f8bb00f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:27.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 shutdown_connections 2024-01-27T23:28:27.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:27.238+0000 7f8bb6312700 1 -- 172.21.15.194:0/2527193828 wait complete. 2024-01-27T23:28:27.230 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:27.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:27.237869+0000 mon.smithi121 (mon.0) 180 : audit [DBG] from='client.? 172.21.15.194:0/2527193828' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:28.603 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:28.603 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:28.837 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:27.285546+0000 mgr.smithi121.hcppvz (mgr.14162) 96 : cluster [DBG] pgmap v89: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:29.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.780+0000 7fd3e7089700 1 -- 172.21.15.194:0/901352863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 msgr2=0x7fd3e00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:29.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.780+0000 7fd3e7089700 1 --2- 172.21.15.194:0/901352863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e00f5830 secure :-1 s=READY pgs=203 cs=0 l=1 rev1=1 rx=0x7fd3d4009fb0 tx=0x7fd3d4009b30).stop 2024-01-27T23:28:29.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.780+0000 7fd3e7089700 1 -- 172.21.15.194:0/901352863 shutdown_connections 2024-01-27T23:28:29.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.780+0000 7fd3e7089700 1 --2- 172.21.15.194:0/901352863 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e00f5830 unknown :-1 s=CLOSED pgs=203 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:29.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.780+0000 7fd3e7089700 1 -- 172.21.15.194:0/901352863 >> 172.21.15.194:0/901352863 conn(0x7fd3e00f0b70 msgr2=0x7fd3e00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:29.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.780+0000 7fd3e7089700 1 -- 172.21.15.194:0/901352863 shutdown_connections 2024-01-27T23:28:29.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.780+0000 7fd3e7089700 1 -- 172.21.15.194:0/901352863 wait complete. 2024-01-27T23:28:29.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.781+0000 7fd3e7089700 1 Processor -- start 2024-01-27T23:28:29.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.781+0000 7fd3e7089700 1 -- start start 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.781+0000 7fd3e7089700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e0109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.781+0000 7fd3e7089700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd3e0060240 con 0x7fd3e00f5410 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.781+0000 7fd3e4e25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e0109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3e4e25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e0109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48036/0 (socket says 172.21.15.194:48036) 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3e4e25700 1 -- 172.21.15.194:0/1735154021 learned_addr learned my addr 172.21.15.194:0/1735154021 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3e4e25700 1 -- 172.21.15.194:0/1735154021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd3d4005040 con 0x7fd3e00f5410 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3e4e25700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e0109430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd3d4013510 tx=0x7fd3d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3dd7fa700 1 -- 172.21.15.194:0/1735154021 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3d4009320 con 0x7fd3e00f5410 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd3e0060440 con 0x7fd3e00f5410 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3dd7fa700 1 -- 172.21.15.194:0/1735154021 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd3d4013f00 con 0x7fd3e00f5410 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3dd7fa700 1 -- 172.21.15.194:0/1735154021 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd3d4014d90 con 0x7fd3e00f5410 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.782+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd3e0109ae0 con 0x7fd3e00f5410 2024-01-27T23:28:29.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.783+0000 7fd3dd7fa700 1 -- 172.21.15.194:0/1735154021 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd3d40263a0 con 0x7fd3e00f5410 2024-01-27T23:28:29.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.783+0000 7fd3dd7fa700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd3d002ff60 0x7fd3d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:29.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.783+0000 7fd3dd7fa700 1 -- 172.21.15.194:0/1735154021 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd3d4041020 con 0x7fd3e00f5410 2024-01-27T23:28:29.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.783+0000 7fd3dffff700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd3d002ff60 0x7fd3d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:29.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.783+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd3e0059100 con 0x7fd3e00f5410 2024-01-27T23:28:29.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.784+0000 7fd3dffff700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd3d002ff60 0x7fd3d0032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd3cc0096a0 tx=0x7fd3cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:29.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.788+0000 7fd3dd7fa700 1 -- 172.21.15.194:0/1735154021 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd3d4012920 con 0x7fd3e00f5410 2024-01-27T23:28:29.984 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.993+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd3e0045ad0 con 0x7fd3e00f5410 2024-01-27T23:28:29.984 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.994+0000 7fd3dd7fa700 1 -- 172.21.15.194:0/1735154021 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd3d402b020 con 0x7fd3e00f5410 2024-01-27T23:28:29.984 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:29.984 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:29.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd3d002ff60 msgr2=0x7fd3d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:29.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd3d002ff60 0x7fd3d0032420 secure :-1 s=READY pgs=68 cs=0 l=1 rev1=1 rx=0x7fd3cc0096a0 tx=0x7fd3cc006b40).stop 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 msgr2=0x7fd3e0109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e0109430 secure :-1 s=READY pgs=204 cs=0 l=1 rev1=1 rx=0x7fd3d4013510 tx=0x7fd3d400b040).stop 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 shutdown_connections 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd3d002ff60 0x7fd3d0032420 unknown :-1 s=CLOSED pgs=68 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 --2- 172.21.15.194:0/1735154021 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd3e00f5410 0x7fd3e0109430 unknown :-1 s=CLOSED pgs=204 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 >> 172.21.15.194:0/1735154021 conn(0x7fd3e00f0b70 msgr2=0x7fd3e00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.995+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 shutdown_connections 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:29.996+0000 7fd3e7089700 1 -- 172.21.15.194:0/1735154021 wait complete. 2024-01-27T23:28:29.987 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:30.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:29.285849+0000 mgr.smithi121.hcppvz (mgr.14162) 97 : cluster [DBG] pgmap v90: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:30.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:29.994858+0000 mon.smithi121 (mon.0) 181 : audit [DBG] from='client.? 172.21.15.194:0/1735154021' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:31.361 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:31.361 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:32.547 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 -- 172.21.15.194:0/105077407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 msgr2=0x7f44f40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:32.547 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 --2- 172.21.15.194:0/105077407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f40f5830 secure :-1 s=READY pgs=205 cs=0 l=1 rev1=1 rx=0x7f44e8009fb0 tx=0x7f44e8009b30).stop 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 -- 172.21.15.194:0/105077407 shutdown_connections 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 --2- 172.21.15.194:0/105077407 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f40f5830 unknown :-1 s=CLOSED pgs=205 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 -- 172.21.15.194:0/105077407 >> 172.21.15.194:0/105077407 conn(0x7f44f40f0b70 msgr2=0x7f44f40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 -- 172.21.15.194:0/105077407 shutdown_connections 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 -- 172.21.15.194:0/105077407 wait complete. 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 Processor -- start 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.556+0000 7f44fb071700 1 -- start start 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.557+0000 7f44fb071700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:32.548 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.557+0000 7f44fb071700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f44f4060240 con 0x7f44f40f5410 2024-01-27T23:28:32.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.557+0000 7f44f8e0d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:32.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.557+0000 7f44f8e0d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48062/0 (socket says 172.21.15.194:48062) 2024-01-27T23:28:32.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.557+0000 7f44f8e0d700 1 -- 172.21.15.194:0/3998027899 learned_addr learned my addr 172.21.15.194:0/3998027899 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:32.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.557+0000 7f44f8e0d700 1 -- 172.21.15.194:0/3998027899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f44e8005040 con 0x7f44f40f5410 2024-01-27T23:28:32.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.558+0000 7f44f8e0d700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f4109430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f44e8013510 tx=0x7f44e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:32.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.558+0000 7f44f17fa700 1 -- 172.21.15.194:0/3998027899 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e8009320 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.558+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f44f4060440 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.558+0000 7f44f17fa700 1 -- 172.21.15.194:0/3998027899 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f44e8013f00 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.558+0000 7f44f17fa700 1 -- 172.21.15.194:0/3998027899 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f44e8014d90 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.558+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f44f4109ae0 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.559+0000 7f44f17fa700 1 -- 172.21.15.194:0/3998027899 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f44e80263a0 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.559+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f44f4045ad0 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.559+0000 7f44f17fa700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f44e402ffb0 0x7f44e4032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.559+0000 7f44f17fa700 1 -- 172.21.15.194:0/3998027899 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f44e8041370 con 0x7f44f40f5410 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.559+0000 7f44f3fff700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f44e402ffb0 0x7f44e4032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:32.551 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.559+0000 7f44f3fff700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f44e402ffb0 0x7f44e4032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f44e00096a0 tx=0x7f44e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:32.554 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.564+0000 7f44f17fa700 1 -- 172.21.15.194:0/3998027899 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f44e80126f0 con 0x7f44f40f5410 2024-01-27T23:28:32.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:31.286268+0000 mgr.smithi121.hcppvz (mgr.14162) 98 : cluster [DBG] pgmap v91: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:32.763 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.773+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f44f40fae60 con 0x7f44f40f5410 2024-01-27T23:28:32.764 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.773+0000 7f44f17fa700 1 -- 172.21.15.194:0/3998027899 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f44e8012330 con 0x7f44f40f5410 2024-01-27T23:28:32.764 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:32.764 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:32.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f44e402ffb0 msgr2=0x7f44e4032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:32.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f44e402ffb0 0x7f44e4032470 secure :-1 s=READY pgs=69 cs=0 l=1 rev1=1 rx=0x7f44e00096a0 tx=0x7f44e0006b40).stop 2024-01-27T23:28:32.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 msgr2=0x7f44f4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:32.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f4109430 secure :-1 s=READY pgs=206 cs=0 l=1 rev1=1 rx=0x7f44e8013510 tx=0x7f44e800b040).stop 2024-01-27T23:28:32.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 shutdown_connections 2024-01-27T23:28:32.766 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f44e402ffb0 0x7f44e4032470 unknown :-1 s=CLOSED pgs=69 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:32.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 --2- 172.21.15.194:0/3998027899 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f44f40f5410 0x7f44f4109430 unknown :-1 s=CLOSED pgs=206 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:32.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 >> 172.21.15.194:0/3998027899 conn(0x7f44f40f0b70 msgr2=0x7f44f40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:32.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 shutdown_connections 2024-01-27T23:28:32.767 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:32.775+0000 7f44fb071700 1 -- 172.21.15.194:0/3998027899 wait complete. 2024-01-27T23:28:32.767 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:33.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:33 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:32.774638+0000 mon.smithi121 (mon.0) 182 : audit [DBG] from='client.? 172.21.15.194:0/3998027899' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:34.166 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:34.166 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:34.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:33.286559+0000 mgr.smithi121.hcppvz (mgr.14162) 99 : cluster [DBG] pgmap v92: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:35.321 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.329+0000 7fa3d37b3700 1 -- 172.21.15.194:0/4284008205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 msgr2=0x7fa3cc0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.329+0000 7fa3d37b3700 1 --2- 172.21.15.194:0/4284008205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f90f0 secure :-1 s=READY pgs=207 cs=0 l=1 rev1=1 rx=0x7fa3bc009fb0 tx=0x7fa3bc009b30).stop 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.329+0000 7fa3d37b3700 1 -- 172.21.15.194:0/4284008205 shutdown_connections 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.329+0000 7fa3d37b3700 1 --2- 172.21.15.194:0/4284008205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f90f0 unknown :-1 s=CLOSED pgs=207 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.329+0000 7fa3d37b3700 1 -- 172.21.15.194:0/4284008205 >> 172.21.15.194:0/4284008205 conn(0x7fa3cc0f0b70 msgr2=0x7fa3cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.329+0000 7fa3d37b3700 1 -- 172.21.15.194:0/4284008205 shutdown_connections 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.329+0000 7fa3d37b3700 1 -- 172.21.15.194:0/4284008205 wait complete. 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.330+0000 7fa3d37b3700 1 Processor -- start 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.330+0000 7fa3d37b3700 1 -- start start 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.330+0000 7fa3d37b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.330+0000 7fa3d37b3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa3cc0f4ff0 con 0x7fa3cc060280 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.330+0000 7fa3d154f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:35.322 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.330+0000 7fa3d154f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32946/0 (socket says 172.21.15.194:32946) 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.330+0000 7fa3d154f700 1 -- 172.21.15.194:0/1007465618 learned_addr learned my addr 172.21.15.194:0/1007465618 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.331+0000 7fa3d154f700 1 -- 172.21.15.194:0/1007465618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa3bc005040 con 0x7fa3cc060280 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.331+0000 7fa3d154f700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f6b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fa3bc013510 tx=0x7fa3bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.331+0000 7fa3c1ffb700 1 -- 172.21.15.194:0/1007465618 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3bc009320 con 0x7fa3cc060280 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.331+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa3cc0f5190 con 0x7fa3cc060280 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.331+0000 7fa3c1ffb700 1 -- 172.21.15.194:0/1007465618 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa3bc013f00 con 0x7fa3cc060280 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.331+0000 7fa3c1ffb700 1 -- 172.21.15.194:0/1007465618 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa3bc014dc0 con 0x7fa3cc060280 2024-01-27T23:28:35.323 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.331+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa3cc0f5620 con 0x7fa3cc060280 2024-01-27T23:28:35.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.332+0000 7fa3c1ffb700 1 -- 172.21.15.194:0/1007465618 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa3bc014420 con 0x7fa3cc060280 2024-01-27T23:28:35.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.332+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa3cc059100 con 0x7fa3cc060280 2024-01-27T23:28:35.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.332+0000 7fa3c1ffb700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa3b802ff60 0x7fa3b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:35.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.332+0000 7fa3c1ffb700 1 -- 172.21.15.194:0/1007465618 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa3bc0410f0 con 0x7fa3cc060280 2024-01-27T23:28:35.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.332+0000 7fa3d0d4e700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa3b802ff60 0x7fa3b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:35.324 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.333+0000 7fa3d0d4e700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa3b802ff60 0x7fa3b8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fa3c80096a0 tx=0x7fa3c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:35.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.336+0000 7fa3c1ffb700 1 -- 172.21.15.194:0/1007465618 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa3bc0095f0 con 0x7fa3cc060280 2024-01-27T23:28:35.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.541+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa3cc0fbf50 con 0x7fa3cc060280 2024-01-27T23:28:35.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.541+0000 7fa3c1ffb700 1 -- 172.21.15.194:0/1007465618 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa3bc01cb10 con 0x7fa3cc060280 2024-01-27T23:28:35.532 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:35.533 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa3b802ff60 msgr2=0x7fa3b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa3b802ff60 0x7fa3b8032420 secure :-1 s=READY pgs=70 cs=0 l=1 rev1=1 rx=0x7fa3c80096a0 tx=0x7fa3c8006b40).stop 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 msgr2=0x7fa3cc0f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f6b70 secure :-1 s=READY pgs=208 cs=0 l=1 rev1=1 rx=0x7fa3bc013510 tx=0x7fa3bc00b040).stop 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 shutdown_connections 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa3b802ff60 0x7fa3b8032420 unknown :-1 s=CLOSED pgs=70 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 --2- 172.21.15.194:0/1007465618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa3cc060280 0x7fa3cc0f6b70 unknown :-1 s=CLOSED pgs=208 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 >> 172.21.15.194:0/1007465618 conn(0x7fa3cc0f0b70 msgr2=0x7fa3cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:35.534 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 shutdown_connections 2024-01-27T23:28:35.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:35.543+0000 7fa3d37b3700 1 -- 172.21.15.194:0/1007465618 wait complete. 2024-01-27T23:28:35.535 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:36.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:35.286911+0000 mgr.smithi121.hcppvz (mgr.14162) 100 : cluster [DBG] pgmap v93: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:36.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:35.542711+0000 mon.smithi121 (mon.0) 183 : audit [DBG] from='client.? 172.21.15.194:0/1007465618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:36.909 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:36.909 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:38.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.048+0000 7f2867adc700 1 -- 172.21.15.194:0/3374575117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 msgr2=0x7f28600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:38.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.048+0000 7f2867adc700 1 --2- 172.21.15.194:0/3374575117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f28600f5830 secure :-1 s=READY pgs=209 cs=0 l=1 rev1=1 rx=0x7f2850009fb0 tx=0x7f2850009b30).stop 2024-01-27T23:28:38.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.048+0000 7f2867adc700 1 -- 172.21.15.194:0/3374575117 shutdown_connections 2024-01-27T23:28:38.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.048+0000 7f2867adc700 1 --2- 172.21.15.194:0/3374575117 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f28600f5830 unknown :-1 s=CLOSED pgs=209 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:38.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.048+0000 7f2867adc700 1 -- 172.21.15.194:0/3374575117 >> 172.21.15.194:0/3374575117 conn(0x7f28600f0b70 msgr2=0x7f28600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:38.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.048+0000 7f2867adc700 1 -- 172.21.15.194:0/3374575117 shutdown_connections 2024-01-27T23:28:38.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.048+0000 7f2867adc700 1 -- 172.21.15.194:0/3374575117 wait complete. 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2867adc700 1 Processor -- start 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2867adc700 1 -- start start 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2867adc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f2860109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2867adc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2860060220 con 0x7f28600f5410 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2865878700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f2860109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2865878700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f2860109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32958/0 (socket says 172.21.15.194:32958) 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2865878700 1 -- 172.21.15.194:0/3140569850 learned_addr learned my addr 172.21.15.194:0/3140569850 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.049+0000 7f2865878700 1 -- 172.21.15.194:0/3140569850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2850005040 con 0x7f28600f5410 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.050+0000 7f2865878700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f2860109450 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f2850013510 tx=0x7f285000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:38.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.050+0000 7f28567fc700 1 -- 172.21.15.194:0/3140569850 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2850009320 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.050+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2860060420 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.050+0000 7f28567fc700 1 -- 172.21.15.194:0/3140569850 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2850013f00 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.050+0000 7f28567fc700 1 -- 172.21.15.194:0/3140569850 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2850014d90 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.050+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2860109b00 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.050+0000 7f28567fc700 1 -- 172.21.15.194:0/3140569850 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f28500095f0 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.051+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2860059100 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.051+0000 7f28567fc700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f284c02ff60 0x7f284c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.051+0000 7f28567fc700 1 -- 172.21.15.194:0/3140569850 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28500410a0 con 0x7f28600f5410 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.051+0000 7f2865077700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f284c02ff60 0x7f284c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:38.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.051+0000 7f2865077700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f284c02ff60 0x7f284c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f285c0096a0 tx=0x7f285c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:38.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.056+0000 7f28567fc700 1 -- 172.21.15.194:0/3140569850 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f28500268c0 con 0x7f28600f5410 2024-01-27T23:28:38.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.263+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2860045ad0 con 0x7f28600f5410 2024-01-27T23:28:38.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.264+0000 7f28567fc700 1 -- 172.21.15.194:0/3140569850 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2850010230 con 0x7f28600f5410 2024-01-27T23:28:38.255 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:38.255 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:38.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f284c02ff60 msgr2=0x7f284c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:38.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f284c02ff60 0x7f284c032420 secure :-1 s=READY pgs=71 cs=0 l=1 rev1=1 rx=0x7f285c0096a0 tx=0x7f285c006b40).stop 2024-01-27T23:28:38.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 msgr2=0x7f2860109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:38.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f2860109450 secure :-1 s=READY pgs=210 cs=0 l=1 rev1=1 rx=0x7f2850013510 tx=0x7f285000b040).stop 2024-01-27T23:28:38.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 shutdown_connections 2024-01-27T23:28:38.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f284c02ff60 0x7f284c032420 unknown :-1 s=CLOSED pgs=71 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:38.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 --2- 172.21.15.194:0/3140569850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28600f5410 0x7f2860109450 unknown :-1 s=CLOSED pgs=210 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:38.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.265+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 >> 172.21.15.194:0/3140569850 conn(0x7f28600f0b70 msgr2=0x7f28600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:38.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.266+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 shutdown_connections 2024-01-27T23:28:38.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:38.266+0000 7f2867adc700 1 -- 172.21.15.194:0/3140569850 wait complete. 2024-01-27T23:28:38.257 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:38.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:37.287250+0000 mgr.smithi121.hcppvz (mgr.14162) 101 : cluster [DBG] pgmap v94: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:38.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:38.265224+0000 mon.smithi121 (mon.0) 184 : audit [DBG] from='client.? 172.21.15.194:0/3140569850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:39.627 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:39.627 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:40.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:39.287636+0000 mgr.smithi121.hcppvz (mgr.14162) 102 : cluster [DBG] pgmap v95: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:40.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.778+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2354836676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 msgr2=0x7f93a40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:40.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.778+0000 7f93ab8a8700 1 --2- 172.21.15.194:0/2354836676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a40f9150 secure :-1 s=READY pgs=211 cs=0 l=1 rev1=1 rx=0x7f9394009fb0 tx=0x7f9394009b30).stop 2024-01-27T23:28:40.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.778+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2354836676 shutdown_connections 2024-01-27T23:28:40.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.778+0000 7f93ab8a8700 1 --2- 172.21.15.194:0/2354836676 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a40f9150 unknown :-1 s=CLOSED pgs=211 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:40.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.778+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2354836676 >> 172.21.15.194:0/2354836676 conn(0x7f93a40f0b70 msgr2=0x7f93a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.778+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2354836676 shutdown_connections 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2354836676 wait complete. 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93ab8a8700 1 Processor -- start 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93ab8a8700 1 -- start start 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93ab8a8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a41066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93ab8a8700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f93a4106b90 con 0x7f93a4060220 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93a9644700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a41066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93a9644700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a41066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32982/0 (socket says 172.21.15.194:32982) 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.779+0000 7f93a9644700 1 -- 172.21.15.194:0/2043505156 learned_addr learned my addr 172.21.15.194:0/2043505156 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.780+0000 7f93a9644700 1 -- 172.21.15.194:0/2043505156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9394005040 con 0x7f93a4060220 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.780+0000 7f93a9644700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a41066a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9394013510 tx=0x7f939400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.780+0000 7f9399ffb700 1 -- 172.21.15.194:0/2043505156 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394009320 con 0x7f93a4060220 2024-01-27T23:28:40.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.780+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f93a4106d90 con 0x7f93a4060220 2024-01-27T23:28:40.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.780+0000 7f9399ffb700 1 -- 172.21.15.194:0/2043505156 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9394013f00 con 0x7f93a4060220 2024-01-27T23:28:40.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.780+0000 7f9399ffb700 1 -- 172.21.15.194:0/2043505156 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9394014dc0 con 0x7f93a4060220 2024-01-27T23:28:40.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.780+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f93a4107220 con 0x7f93a4060220 2024-01-27T23:28:40.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.781+0000 7f9399ffb700 1 -- 172.21.15.194:0/2043505156 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f93940095f0 con 0x7f93a4060220 2024-01-27T23:28:40.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.781+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f93a4045ad0 con 0x7f93a4060220 2024-01-27T23:28:40.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.781+0000 7f9399ffb700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f939002ff60 0x7f9390032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:40.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.781+0000 7f9399ffb700 1 -- 172.21.15.194:0/2043505156 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9394041020 con 0x7f93a4060220 2024-01-27T23:28:40.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.781+0000 7f93a8e43700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f939002ff60 0x7f9390032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:40.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.782+0000 7f93a8e43700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f939002ff60 0x7f9390032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f93a00096a0 tx=0x7f93a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:40.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.786+0000 7f9399ffb700 1 -- 172.21.15.194:0/2043505156 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9394014420 con 0x7f93a4060220 2024-01-27T23:28:40.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.992+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f93a4108070 con 0x7f93a4060220 2024-01-27T23:28:40.983 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.992+0000 7f9399ffb700 1 -- 172.21.15.194:0/2043505156 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f939401c920 con 0x7f93a4060220 2024-01-27T23:28:40.983 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:40.983 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f939002ff60 msgr2=0x7f9390032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f939002ff60 0x7f9390032420 secure :-1 s=READY pgs=72 cs=0 l=1 rev1=1 rx=0x7f93a00096a0 tx=0x7f93a0006b40).stop 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 msgr2=0x7f93a41066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a41066a0 secure :-1 s=READY pgs=212 cs=0 l=1 rev1=1 rx=0x7f9394013510 tx=0x7f939400b040).stop 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 shutdown_connections 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f939002ff60 0x7f9390032420 unknown :-1 s=CLOSED pgs=72 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 --2- 172.21.15.194:0/2043505156 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f93a4060220 0x7f93a41066a0 unknown :-1 s=CLOSED pgs=212 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 >> 172.21.15.194:0/2043505156 conn(0x7f93a40f0b70 msgr2=0x7f93a40f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 shutdown_connections 2024-01-27T23:28:40.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:40.994+0000 7f93ab8a8700 1 -- 172.21.15.194:0/2043505156 wait complete. 2024-01-27T23:28:40.986 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:41.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:41 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:40.993777+0000 mon.smithi121 (mon.0) 185 : audit [DBG] from='client.? 172.21.15.194:0/2043505156' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:42.355 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:42.356 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:42.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:41.288112+0000 mgr.smithi121.hcppvz (mgr.14162) 103 : cluster [DBG] pgmap v96: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:43.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.535+0000 7f38d37b1700 1 -- 172.21.15.194:0/29357618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 msgr2=0x7f38cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:43.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.535+0000 7f38d37b1700 1 --2- 172.21.15.194:0/29357618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc0f5830 secure :-1 s=READY pgs=213 cs=0 l=1 rev1=1 rx=0x7f38c0009fb0 tx=0x7f38c0009b30).stop 2024-01-27T23:28:43.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.535+0000 7f38d37b1700 1 -- 172.21.15.194:0/29357618 shutdown_connections 2024-01-27T23:28:43.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.535+0000 7f38d37b1700 1 --2- 172.21.15.194:0/29357618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc0f5830 unknown :-1 s=CLOSED pgs=213 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:43.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.535+0000 7f38d37b1700 1 -- 172.21.15.194:0/29357618 >> 172.21.15.194:0/29357618 conn(0x7f38cc0f0b70 msgr2=0x7f38cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:43.527 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.535+0000 7f38d37b1700 1 -- 172.21.15.194:0/29357618 shutdown_connections 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.535+0000 7f38d37b1700 1 -- 172.21.15.194:0/29357618 wait complete. 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d37b1700 1 Processor -- start 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d37b1700 1 -- start start 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d37b1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d37b1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f38cc105b20 con 0x7f38cc0f5410 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d154d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d154d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32994/0 (socket says 172.21.15.194:32994) 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d154d700 1 -- 172.21.15.194:0/3928379383 learned_addr learned my addr 172.21.15.194:0/3928379383 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d154d700 1 -- 172.21.15.194:0/3928379383 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f38c0005040 con 0x7f38cc0f5410 2024-01-27T23:28:43.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.536+0000 7f38d154d700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f38c0013510 tx=0x7f38c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:43.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.537+0000 7f38bdffb700 1 -- 172.21.15.194:0/3928379383 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c0009320 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.537+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f38cc105d20 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.537+0000 7f38bdffb700 1 -- 172.21.15.194:0/3928379383 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f38c0013f00 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.537+0000 7f38bdffb700 1 -- 172.21.15.194:0/3928379383 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f38c0014d90 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.537+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f38cc1061b0 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.538+0000 7f38bdffb700 1 -- 172.21.15.194:0/3928379383 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f38c00095f0 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.538+0000 7f38bdffb700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f38b802ff60 0x7f38b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.538+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f38cc045ad0 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.538+0000 7f38bdffb700 1 -- 172.21.15.194:0/3928379383 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f38c0040fc0 con 0x7f38cc0f5410 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.538+0000 7f38d0d4c700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f38b802ff60 0x7f38b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:43.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.538+0000 7f38d0d4c700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f38b802ff60 0x7f38b8032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f38c80096a0 tx=0x7f38c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:43.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.543+0000 7f38bdffb700 1 -- 172.21.15.194:0/3928379383 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f38c0014420 con 0x7f38cc0f5410 2024-01-27T23:28:43.744 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.754+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f38cc0faf50 con 0x7f38cc0f5410 2024-01-27T23:28:43.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.754+0000 7f38bdffb700 1 -- 172.21.15.194:0/3928379383 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f38c002c090 con 0x7f38cc0f5410 2024-01-27T23:28:43.745 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:43.746 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f38b802ff60 msgr2=0x7f38b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f38b802ff60 0x7f38b8032420 secure :-1 s=READY pgs=73 cs=0 l=1 rev1=1 rx=0x7f38c80096a0 tx=0x7f38c8006b40).stop 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 msgr2=0x7f38cc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc105630 secure :-1 s=READY pgs=214 cs=0 l=1 rev1=1 rx=0x7f38c0013510 tx=0x7f38c000b040).stop 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 shutdown_connections 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f38b802ff60 0x7f38b8032420 unknown :-1 s=CLOSED pgs=73 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 --2- 172.21.15.194:0/3928379383 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f38cc0f5410 0x7f38cc105630 unknown :-1 s=CLOSED pgs=214 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 >> 172.21.15.194:0/3928379383 conn(0x7f38cc0f0b70 msgr2=0x7f38cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 shutdown_connections 2024-01-27T23:28:43.747 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:43.756+0000 7f38d37b1700 1 -- 172.21.15.194:0/3928379383 wait complete. 2024-01-27T23:28:43.748 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:44.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:43.288553+0000 mgr.smithi121.hcppvz (mgr.14162) 104 : cluster [DBG] pgmap v97: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:44.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:43.755679+0000 mon.smithi121 (mon.0) 186 : audit [DBG] from='client.? 172.21.15.194:0/3928379383' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:45.117 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:45.118 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.259+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1122872646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 msgr2=0x7fcc2c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.259+0000 7fcc30dd7700 1 --2- 172.21.15.194:0/1122872646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c0f5830 secure :-1 s=READY pgs=215 cs=0 l=1 rev1=1 rx=0x7fcc14009fb0 tx=0x7fcc14009b30).stop 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.259+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1122872646 shutdown_connections 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.259+0000 7fcc30dd7700 1 --2- 172.21.15.194:0/1122872646 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c0f5830 unknown :-1 s=CLOSED pgs=215 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.259+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1122872646 >> 172.21.15.194:0/1122872646 conn(0x7fcc2c0f0b70 msgr2=0x7fcc2c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.259+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1122872646 shutdown_connections 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.259+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1122872646 wait complete. 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.260+0000 7fcc30dd7700 1 Processor -- start 2024-01-27T23:28:46.251 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.260+0000 7fcc30dd7700 1 -- start start 2024-01-27T23:28:46.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.260+0000 7fcc30dd7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:46.252 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.260+0000 7fcc30dd7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcc2c105b20 con 0x7fcc2c0f5410 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.260+0000 7fcc2a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.260+0000 7fcc2a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55936/0 (socket says 172.21.15.194:55936) 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc2a59c700 1 -- 172.21.15.194:0/1999710053 learned_addr learned my addr 172.21.15.194:0/1999710053 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc2a59c700 1 -- 172.21.15.194:0/1999710053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcc14005040 con 0x7fcc2c0f5410 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc2a59c700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcc14014000 tx=0x7fcc1400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc22ffd700 1 -- 172.21.15.194:0/1999710053 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc140095f0 con 0x7fcc2c0f5410 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcc2c105d20 con 0x7fcc2c0f5410 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc22ffd700 1 -- 172.21.15.194:0/1999710053 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcc14007e70 con 0x7fcc2c0f5410 2024-01-27T23:28:46.254 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc22ffd700 1 -- 172.21.15.194:0/1999710053 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcc14014ea0 con 0x7fcc2c0f5410 2024-01-27T23:28:46.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.261+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcc2c1061b0 con 0x7fcc2c0f5410 2024-01-27T23:28:46.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.262+0000 7fcc22ffd700 1 -- 172.21.15.194:0/1999710053 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcc14027070 con 0x7fcc2c0f5410 2024-01-27T23:28:46.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.262+0000 7fcc22ffd700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc1802ff60 0x7fcc18032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:46.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.262+0000 7fcc22ffd700 1 -- 172.21.15.194:0/1999710053 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcc140426d0 con 0x7fcc2c0f5410 2024-01-27T23:28:46.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.262+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcc2c045ad0 con 0x7fcc2c0f5410 2024-01-27T23:28:46.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.262+0000 7fcc29d9b700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc1802ff60 0x7fcc18032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:46.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.263+0000 7fcc29d9b700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc1802ff60 0x7fcc18032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcc1c0096a0 tx=0x7fcc1c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:46.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.267+0000 7fcc22ffd700 1 -- 172.21.15.194:0/1999710053 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcc1401f3c0 con 0x7fcc2c0f5410 2024-01-27T23:28:46.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.472+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcc2c106ed0 con 0x7fcc2c0f5410 2024-01-27T23:28:46.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.473+0000 7fcc22ffd700 1 -- 172.21.15.194:0/1999710053 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcc1402f090 con 0x7fcc2c0f5410 2024-01-27T23:28:46.464 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:46.464 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:46.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc1802ff60 msgr2=0x7fcc18032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:46.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc1802ff60 0x7fcc18032420 secure :-1 s=READY pgs=74 cs=0 l=1 rev1=1 rx=0x7fcc1c0096a0 tx=0x7fcc1c006b40).stop 2024-01-27T23:28:46.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 msgr2=0x7fcc2c105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:46.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c105630 secure :-1 s=READY pgs=216 cs=0 l=1 rev1=1 rx=0x7fcc14014000 tx=0x7fcc1400b040).stop 2024-01-27T23:28:46.465 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 shutdown_connections 2024-01-27T23:28:46.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc1802ff60 0x7fcc18032420 unknown :-1 s=CLOSED pgs=74 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:46.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 --2- 172.21.15.194:0/1999710053 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcc2c0f5410 0x7fcc2c105630 unknown :-1 s=CLOSED pgs=216 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:46.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 >> 172.21.15.194:0/1999710053 conn(0x7fcc2c0f0b70 msgr2=0x7fcc2c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:46.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 shutdown_connections 2024-01-27T23:28:46.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:46.474+0000 7fcc30dd7700 1 -- 172.21.15.194:0/1999710053 wait complete. 2024-01-27T23:28:46.466 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:46.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:45.288949+0000 mgr.smithi121.hcppvz (mgr.14162) 105 : cluster [DBG] pgmap v98: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:47.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:47 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:46.473899+0000 mon.smithi121 (mon.0) 187 : audit [DBG] from='client.? 172.21.15.194:0/1999710053' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:47.884 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:47.885 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:48.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:47.289314+0000 mgr.smithi121.hcppvz (mgr.14162) 106 : cluster [DBG] pgmap v99: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.060+0000 7f314cd91700 1 -- 172.21.15.194:0/1263940868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 msgr2=0x7f31480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.060+0000 7f314cd91700 1 --2- 172.21.15.194:0/1263940868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f31480f5830 secure :-1 s=READY pgs=217 cs=0 l=1 rev1=1 rx=0x7f3138009fb0 tx=0x7f3138009b30).stop 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.060+0000 7f314cd91700 1 -- 172.21.15.194:0/1263940868 shutdown_connections 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.060+0000 7f314cd91700 1 --2- 172.21.15.194:0/1263940868 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f31480f5830 unknown :-1 s=CLOSED pgs=217 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.060+0000 7f314cd91700 1 -- 172.21.15.194:0/1263940868 >> 172.21.15.194:0/1263940868 conn(0x7f31480f0b70 msgr2=0x7f31480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.060+0000 7f314cd91700 1 -- 172.21.15.194:0/1263940868 shutdown_connections 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.060+0000 7f314cd91700 1 -- 172.21.15.194:0/1263940868 wait complete. 2024-01-27T23:28:49.053 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314cd91700 1 Processor -- start 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314cd91700 1 -- start start 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314cd91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f3148106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314cd91700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3148106c20 con 0x7f31480f5410 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f3148106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f3148106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55956/0 (socket says 172.21.15.194:55956) 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314659c700 1 -- 172.21.15.194:0/4076777651 learned_addr learned my addr 172.21.15.194:0/4076777651 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.061+0000 7f314659c700 1 -- 172.21.15.194:0/4076777651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3138005040 con 0x7f31480f5410 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.062+0000 7f314659c700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f3148106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f3138013510 tx=0x7f313800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.062+0000 7f3136ffd700 1 -- 172.21.15.194:0/4076777651 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3138009320 con 0x7f31480f5410 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.062+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3148106e20 con 0x7f31480f5410 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.062+0000 7f3136ffd700 1 -- 172.21.15.194:0/4076777651 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3138013f00 con 0x7f31480f5410 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.062+0000 7f3136ffd700 1 -- 172.21.15.194:0/4076777651 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3138014d90 con 0x7f31480f5410 2024-01-27T23:28:49.054 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.062+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f31481072b0 con 0x7f31480f5410 2024-01-27T23:28:49.055 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.063+0000 7f3136ffd700 1 -- 172.21.15.194:0/4076777651 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f31380263a0 con 0x7f31480f5410 2024-01-27T23:28:49.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.063+0000 7f3136ffd700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f313002ff60 0x7f3130032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:49.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.063+0000 7f3136ffd700 1 -- 172.21.15.194:0/4076777651 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f31380410c0 con 0x7f31480f5410 2024-01-27T23:28:49.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.063+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3148045ad0 con 0x7f31480f5410 2024-01-27T23:28:49.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.063+0000 7f3145d9b700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f313002ff60 0x7f3130032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:49.056 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.063+0000 7f3145d9b700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f313002ff60 0x7f3130032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f313c0096a0 tx=0x7f313c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:49.058 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.068+0000 7f3136ffd700 1 -- 172.21.15.194:0/4076777651 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3138012920 con 0x7f31480f5410 2024-01-27T23:28:49.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.272+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f31480fae60 con 0x7f31480f5410 2024-01-27T23:28:49.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.273+0000 7f3136ffd700 1 -- 172.21.15.194:0/4076777651 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f313802b020 con 0x7f31480f5410 2024-01-27T23:28:49.264 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:49.264 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:49.265 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f313002ff60 msgr2=0x7f3130032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f313002ff60 0x7f3130032420 secure :-1 s=READY pgs=75 cs=0 l=1 rev1=1 rx=0x7f313c0096a0 tx=0x7f313c006b40).stop 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 msgr2=0x7f3148106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f3148106730 secure :-1 s=READY pgs=218 cs=0 l=1 rev1=1 rx=0x7f3138013510 tx=0x7f313800b040).stop 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 shutdown_connections 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f313002ff60 0x7f3130032420 unknown :-1 s=CLOSED pgs=75 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 --2- 172.21.15.194:0/4076777651 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f31480f5410 0x7f3148106730 unknown :-1 s=CLOSED pgs=218 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 >> 172.21.15.194:0/4076777651 conn(0x7f31480f0b70 msgr2=0x7f31480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 shutdown_connections 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:49.274+0000 7f314cd91700 1 -- 172.21.15.194:0/4076777651 wait complete. 2024-01-27T23:28:49.266 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:49.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:49 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:49.273791+0000 mon.smithi121 (mon.0) 188 : audit [DBG] from='client.? 172.21.15.194:0/4076777651' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:50.660 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:50.660 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:50.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:49.289545+0000 mgr.smithi121.hcppvz (mgr.14162) 107 : cluster [DBG] pgmap v100: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:51.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 -- 172.21.15.194:0/3031422592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 msgr2=0x7f1cac0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:51.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 --2- 172.21.15.194:0/3031422592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac0f90f0 secure :-1 s=READY pgs=219 cs=0 l=1 rev1=1 rx=0x7f1ca0009fb0 tx=0x7f1ca0009b30).stop 2024-01-27T23:28:51.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 -- 172.21.15.194:0/3031422592 shutdown_connections 2024-01-27T23:28:51.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 --2- 172.21.15.194:0/3031422592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac0f90f0 unknown :-1 s=CLOSED pgs=219 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:51.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 -- 172.21.15.194:0/3031422592 >> 172.21.15.194:0/3031422592 conn(0x7f1cac0f0b70 msgr2=0x7f1cac0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:51.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 -- 172.21.15.194:0/3031422592 shutdown_connections 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 -- 172.21.15.194:0/3031422592 wait complete. 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.838+0000 7f1cb2f00700 1 Processor -- start 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.839+0000 7f1cb2f00700 1 -- start start 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.839+0000 7f1cb2f00700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.839+0000 7f1cb2f00700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1cac106bd0 con 0x7f1cac060280 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.839+0000 7f1cb0c9c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.839+0000 7f1cb0c9c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55980/0 (socket says 172.21.15.194:55980) 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.839+0000 7f1cb0c9c700 1 -- 172.21.15.194:0/2775880696 learned_addr learned my addr 172.21.15.194:0/2775880696 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:51.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.839+0000 7f1cb0c9c700 1 -- 172.21.15.194:0/2775880696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ca0005040 con 0x7f1cac060280 2024-01-27T23:28:51.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.840+0000 7f1cb0c9c700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac1066e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1ca0013510 tx=0x7f1ca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:51.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.840+0000 7f1ca97fa700 1 -- 172.21.15.194:0/2775880696 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ca0009320 con 0x7f1cac060280 2024-01-27T23:28:51.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.840+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1cac106dd0 con 0x7f1cac060280 2024-01-27T23:28:51.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.840+0000 7f1ca97fa700 1 -- 172.21.15.194:0/2775880696 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ca0013f00 con 0x7f1cac060280 2024-01-27T23:28:51.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.840+0000 7f1ca97fa700 1 -- 172.21.15.194:0/2775880696 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ca0014dc0 con 0x7f1cac060280 2024-01-27T23:28:51.833 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.840+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1cac107260 con 0x7f1cac060280 2024-01-27T23:28:51.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.841+0000 7f1ca97fa700 1 -- 172.21.15.194:0/2775880696 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ca00095f0 con 0x7f1cac060280 2024-01-27T23:28:51.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.841+0000 7f1ca97fa700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1c9c02ff60 0x7f1c9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:51.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.841+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1cac045ad0 con 0x7f1cac060280 2024-01-27T23:28:51.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.841+0000 7f1ca97fa700 1 -- 172.21.15.194:0/2775880696 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ca0041000 con 0x7f1cac060280 2024-01-27T23:28:51.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.841+0000 7f1cabfff700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1c9c02ff60 0x7f1c9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:51.834 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.841+0000 7f1cabfff700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1c9c02ff60 0x7f1c9c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f1c980096a0 tx=0x7f1c98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:51.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:51.846+0000 7f1ca97fa700 1 -- 172.21.15.194:0/2775880696 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ca0010230 con 0x7f1cac060280 2024-01-27T23:28:52.043 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.052+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1cac108110 con 0x7f1cac060280 2024-01-27T23:28:52.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.053+0000 7f1ca97fa700 1 -- 172.21.15.194:0/2775880696 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ca002c090 con 0x7f1cac060280 2024-01-27T23:28:52.045 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:52.045 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1c9c02ff60 msgr2=0x7f1c9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1c9c02ff60 0x7f1c9c032420 secure :-1 s=READY pgs=76 cs=0 l=1 rev1=1 rx=0x7f1c980096a0 tx=0x7f1c98006b40).stop 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 msgr2=0x7f1cac1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac1066e0 secure :-1 s=READY pgs=220 cs=0 l=1 rev1=1 rx=0x7f1ca0013510 tx=0x7f1ca000b040).stop 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 shutdown_connections 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1c9c02ff60 0x7f1c9c032420 unknown :-1 s=CLOSED pgs=76 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 --2- 172.21.15.194:0/2775880696 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1cac060280 0x7f1cac1066e0 unknown :-1 s=CLOSED pgs=220 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 >> 172.21.15.194:0/2775880696 conn(0x7f1cac0f0b70 msgr2=0x7f1cac0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:52.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 shutdown_connections 2024-01-27T23:28:52.047 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:52.055+0000 7f1cb2f00700 1 -- 172.21.15.194:0/2775880696 wait complete. 2024-01-27T23:28:52.047 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:52.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:51.289918+0000 mgr.smithi121.hcppvz (mgr.14162) 108 : cluster [DBG] pgmap v101: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:52.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:52.054466+0000 mon.smithi121 (mon.0) 189 : audit [DBG] from='client.? 172.21.15.194:0/2775880696' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:53.431 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:53.431 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:54.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:53.290378+0000 mgr.smithi121.hcppvz (mgr.14162) 109 : cluster [DBG] pgmap v102: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.685+0000 7fe9783ed700 1 -- 172.21.15.194:0/2881620769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 msgr2=0x7fe9700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.685+0000 7fe9783ed700 1 --2- 172.21.15.194:0/2881620769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe9700f5830 secure :-1 s=READY pgs=221 cs=0 l=1 rev1=1 rx=0x7fe964009fb0 tx=0x7fe964009b30).stop 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.685+0000 7fe9783ed700 1 -- 172.21.15.194:0/2881620769 shutdown_connections 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.685+0000 7fe9783ed700 1 --2- 172.21.15.194:0/2881620769 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe9700f5830 unknown :-1 s=CLOSED pgs=221 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.685+0000 7fe9783ed700 1 -- 172.21.15.194:0/2881620769 >> 172.21.15.194:0/2881620769 conn(0x7fe9700f0b70 msgr2=0x7fe9700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.685+0000 7fe9783ed700 1 -- 172.21.15.194:0/2881620769 shutdown_connections 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.685+0000 7fe9783ed700 1 -- 172.21.15.194:0/2881620769 wait complete. 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.686+0000 7fe9783ed700 1 Processor -- start 2024-01-27T23:28:54.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.686+0000 7fe9783ed700 1 -- start start 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.686+0000 7fe9783ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe970106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.686+0000 7fe9783ed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe970106c20 con 0x7fe9700f5410 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.686+0000 7fe976189700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe970106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.686+0000 7fe976189700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe970106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:56000/0 (socket says 172.21.15.194:56000) 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.686+0000 7fe976189700 1 -- 172.21.15.194:0/2823741628 learned_addr learned my addr 172.21.15.194:0/2823741628 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.687+0000 7fe976189700 1 -- 172.21.15.194:0/2823741628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe964005040 con 0x7fe9700f5410 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.687+0000 7fe976189700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe970106730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fe964013510 tx=0x7fe96400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.687+0000 7fe962ffd700 1 -- 172.21.15.194:0/2823741628 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe964009320 con 0x7fe9700f5410 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.687+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe970106e20 con 0x7fe9700f5410 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.687+0000 7fe962ffd700 1 -- 172.21.15.194:0/2823741628 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe964013f00 con 0x7fe9700f5410 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.687+0000 7fe962ffd700 1 -- 172.21.15.194:0/2823741628 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe964014d90 con 0x7fe9700f5410 2024-01-27T23:28:54.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.687+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe9701072b0 con 0x7fe9700f5410 2024-01-27T23:28:54.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.688+0000 7fe962ffd700 1 -- 172.21.15.194:0/2823741628 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9640095f0 con 0x7fe9700f5410 2024-01-27T23:28:54.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.688+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe970045ad0 con 0x7fe9700f5410 2024-01-27T23:28:54.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.688+0000 7fe962ffd700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe95c02ff60 0x7fe95c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:54.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.688+0000 7fe962ffd700 1 -- 172.21.15.194:0/2823741628 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe964041080 con 0x7fe9700f5410 2024-01-27T23:28:54.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.688+0000 7fe975988700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe95c02ff60 0x7fe95c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:54.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.688+0000 7fe975988700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe95c02ff60 0x7fe95c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe96c0096a0 tx=0x7fe96c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:54.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.692+0000 7fe962ffd700 1 -- 172.21.15.194:0/2823741628 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe964014420 con 0x7fe9700f5410 2024-01-27T23:28:54.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.895+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe9700fae60 con 0x7fe9700f5410 2024-01-27T23:28:54.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.895+0000 7fe962ffd700 1 -- 172.21.15.194:0/2823741628 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe96401d7f0 con 0x7fe9700f5410 2024-01-27T23:28:54.887 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:54.887 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe95c02ff60 msgr2=0x7fe95c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe95c02ff60 0x7fe95c032420 secure :-1 s=READY pgs=77 cs=0 l=1 rev1=1 rx=0x7fe96c0096a0 tx=0x7fe96c006b40).stop 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 msgr2=0x7fe970106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe970106730 secure :-1 s=READY pgs=222 cs=0 l=1 rev1=1 rx=0x7fe964013510 tx=0x7fe96400b040).stop 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 shutdown_connections 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe95c02ff60 0x7fe95c032420 unknown :-1 s=CLOSED pgs=77 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 --2- 172.21.15.194:0/2823741628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe9700f5410 0x7fe970106730 unknown :-1 s=CLOSED pgs=222 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 >> 172.21.15.194:0/2823741628 conn(0x7fe9700f0b70 msgr2=0x7fe9700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:54.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 shutdown_connections 2024-01-27T23:28:54.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:54.897+0000 7fe9783ed700 1 -- 172.21.15.194:0/2823741628 wait complete. 2024-01-27T23:28:54.889 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:55.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:55 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:54.896760+0000 mon.smithi121 (mon.0) 190 : audit [DBG] from='client.? 172.21.15.194:0/2823741628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:56.283 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:56.283 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:28:56.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:55.290726+0000 mgr.smithi121.hcppvz (mgr.14162) 110 : cluster [DBG] pgmap v103: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:57.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.393+0000 7f75a4a54700 1 -- 172.21.15.194:0/2339771160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 msgr2=0x7f75a00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:57.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.393+0000 7f75a4a54700 1 --2- 172.21.15.194:0/2339771160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a00f6930 secure :-1 s=READY pgs=223 cs=0 l=1 rev1=1 rx=0x7f7588009fb0 tx=0x7f7588009b30).stop 2024-01-27T23:28:57.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.393+0000 7f75a4a54700 1 -- 172.21.15.194:0/2339771160 shutdown_connections 2024-01-27T23:28:57.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.393+0000 7f75a4a54700 1 --2- 172.21.15.194:0/2339771160 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a00f6930 unknown :-1 s=CLOSED pgs=223 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:57.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.393+0000 7f75a4a54700 1 -- 172.21.15.194:0/2339771160 >> 172.21.15.194:0/2339771160 conn(0x7f75a00f1c70 msgr2=0x7f75a00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:57.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.393+0000 7f75a4a54700 1 -- 172.21.15.194:0/2339771160 shutdown_connections 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.393+0000 7f75a4a54700 1 -- 172.21.15.194:0/2339771160 wait complete. 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.394+0000 7f75a4a54700 1 Processor -- start 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.394+0000 7f75a4a54700 1 -- start start 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.394+0000 7f75a4a54700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.394+0000 7f75a4a54700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75a0106c20 con 0x7f75a00f6510 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.394+0000 7f759e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.394+0000 7f759e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:39864/0 (socket says 172.21.15.194:39864) 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.394+0000 7f759e59c700 1 -- 172.21.15.194:0/4038056977 learned_addr learned my addr 172.21.15.194:0/4038056977 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.395+0000 7f759e59c700 1 -- 172.21.15.194:0/4038056977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7588005040 con 0x7f75a00f6510 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.395+0000 7f759e59c700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a0106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7588013510 tx=0x7f758800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.395+0000 7f7596ffd700 1 -- 172.21.15.194:0/4038056977 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7588009320 con 0x7f75a00f6510 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.395+0000 7f7596ffd700 1 -- 172.21.15.194:0/4038056977 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7588013f00 con 0x7f75a00f6510 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.395+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75a0106e20 con 0x7f75a00f6510 2024-01-27T23:28:57.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.395+0000 7f7596ffd700 1 -- 172.21.15.194:0/4038056977 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7588014d90 con 0x7f75a00f6510 2024-01-27T23:28:57.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.395+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75a01072b0 con 0x7f75a00f6510 2024-01-27T23:28:57.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.396+0000 7f7596ffd700 1 -- 172.21.15.194:0/4038056977 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75880095f0 con 0x7f75a00f6510 2024-01-27T23:28:57.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.396+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75a0045ad0 con 0x7f75a00f6510 2024-01-27T23:28:57.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.396+0000 7f7596ffd700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f758c02ff60 0x7f758c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:28:57.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.396+0000 7f7596ffd700 1 -- 172.21.15.194:0/4038056977 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75880410a0 con 0x7f75a00f6510 2024-01-27T23:28:57.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.396+0000 7f759dd9b700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f758c02ff60 0x7f758c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:28:57.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.396+0000 7f759dd9b700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f758c02ff60 0x7f758c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f75900096a0 tx=0x7f7590006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:28:57.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.401+0000 7f7596ffd700 1 -- 172.21.15.194:0/4038056977 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f758801d7f0 con 0x7f75a00f6510 2024-01-27T23:28:57.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.609+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75a00fbf60 con 0x7f75a00f6510 2024-01-27T23:28:57.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.610+0000 7f7596ffd700 1 -- 172.21.15.194:0/4038056977 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7588010230 con 0x7f75a00f6510 2024-01-27T23:28:57.604 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:28:57.604 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:28:57.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f758c02ff60 msgr2=0x7f758c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:57.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f758c02ff60 0x7f758c032420 secure :-1 s=READY pgs=78 cs=0 l=1 rev1=1 rx=0x7f75900096a0 tx=0x7f7590006b40).stop 2024-01-27T23:28:57.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 msgr2=0x7f75a0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:28:57.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a0106730 secure :-1 s=READY pgs=224 cs=0 l=1 rev1=1 rx=0x7f7588013510 tx=0x7f758800b040).stop 2024-01-27T23:28:57.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 shutdown_connections 2024-01-27T23:28:57.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f758c02ff60 0x7f758c032420 unknown :-1 s=CLOSED pgs=78 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:57.605 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 --2- 172.21.15.194:0/4038056977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75a00f6510 0x7f75a0106730 unknown :-1 s=CLOSED pgs=224 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:28:57.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 >> 172.21.15.194:0/4038056977 conn(0x7f75a00f1c70 msgr2=0x7f75a00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:28:57.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 shutdown_connections 2024-01-27T23:28:57.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:28:57.612+0000 7f75a4a54700 1 -- 172.21.15.194:0/4038056977 wait complete. 2024-01-27T23:28:57.606 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:28:58.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:57.291102+0000 mgr.smithi121.hcppvz (mgr.14162) 111 : cluster [DBG] pgmap v104: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:28:58.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:28:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:28:57.611480+0000 mon.smithi121 (mon.0) 191 : audit [DBG] from='client.? 172.21.15.194:0/4038056977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:28:58.981 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:28:58.982 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:00.146 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.153+0000 7ff400506700 1 -- 172.21.15.194:0/2563305608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 msgr2=0x7ff3f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.153+0000 7ff400506700 1 --2- 172.21.15.194:0/2563305608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f80f5830 secure :-1 s=READY pgs=225 cs=0 l=1 rev1=1 rx=0x7ff3f4009fb0 tx=0x7ff3f4009b30).stop 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.153+0000 7ff400506700 1 -- 172.21.15.194:0/2563305608 shutdown_connections 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.153+0000 7ff400506700 1 --2- 172.21.15.194:0/2563305608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f80f5830 unknown :-1 s=CLOSED pgs=225 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.153+0000 7ff400506700 1 -- 172.21.15.194:0/2563305608 >> 172.21.15.194:0/2563305608 conn(0x7ff3f80f0b70 msgr2=0x7ff3f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.153+0000 7ff400506700 1 -- 172.21.15.194:0/2563305608 shutdown_connections 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.153+0000 7ff400506700 1 -- 172.21.15.194:0/2563305608 wait complete. 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.154+0000 7ff400506700 1 Processor -- start 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.154+0000 7ff400506700 1 -- start start 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.154+0000 7ff400506700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f81093c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.154+0000 7ff400506700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff3f8060280 con 0x7ff3f80f5410 2024-01-27T23:29:00.147 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3fe2a2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f81093c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3fe2a2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f81093c0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:39878/0 (socket says 172.21.15.194:39878) 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3fe2a2700 1 -- 172.21.15.194:0/512606568 learned_addr learned my addr 172.21.15.194:0/512606568 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3fe2a2700 1 -- 172.21.15.194:0/512606568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff3f4005040 con 0x7ff3f80f5410 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3fe2a2700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f81093c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7ff3f4013510 tx=0x7ff3f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3eeffd700 1 -- 172.21.15.194:0/512606568 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3f4009320 con 0x7ff3f80f5410 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff3f8060480 con 0x7ff3f80f5410 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3eeffd700 1 -- 172.21.15.194:0/512606568 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff3f4013f00 con 0x7ff3f80f5410 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff3eeffd700 1 -- 172.21.15.194:0/512606568 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff3f4014d90 con 0x7ff3f80f5410 2024-01-27T23:29:00.148 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.155+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff3f8109af0 con 0x7ff3f80f5410 2024-01-27T23:29:00.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.156+0000 7ff3eeffd700 1 -- 172.21.15.194:0/512606568 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff3f40095f0 con 0x7ff3f80f5410 2024-01-27T23:29:00.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.156+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff3f8045ad0 con 0x7ff3f80f5410 2024-01-27T23:29:00.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.156+0000 7ff3eeffd700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff3e402ff60 0x7ff3e4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:00.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.156+0000 7ff3eeffd700 1 -- 172.21.15.194:0/512606568 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff3f4041090 con 0x7ff3f80f5410 2024-01-27T23:29:00.149 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.156+0000 7ff3fdaa1700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff3e402ff60 0x7ff3e4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:00.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.157+0000 7ff3fdaa1700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff3e402ff60 0x7ff3e4032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:00.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.161+0000 7ff3eeffd700 1 -- 172.21.15.194:0/512606568 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff3f4014420 con 0x7ff3f80f5410 2024-01-27T23:29:00.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.370+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff3f810a310 con 0x7ff3f80f5410 2024-01-27T23:29:00.362 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.371+0000 7ff3eeffd700 1 -- 172.21.15.194:0/512606568 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff3f401d7f0 con 0x7ff3f80f5410 2024-01-27T23:29:00.362 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:00.362 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:00.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff3e402ff60 msgr2=0x7ff3e4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:00.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff3e402ff60 0x7ff3e4032420 secure :-1 s=READY pgs=79 cs=0 l=1 rev1=1 rx=0x7ff3e80096a0 tx=0x7ff3e8006b40).stop 2024-01-27T23:29:00.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 msgr2=0x7ff3f81093c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:00.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f81093c0 secure :-1 s=READY pgs=226 cs=0 l=1 rev1=1 rx=0x7ff3f4013510 tx=0x7ff3f400b040).stop 2024-01-27T23:29:00.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 shutdown_connections 2024-01-27T23:29:00.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff3e402ff60 0x7ff3e4032420 unknown :-1 s=CLOSED pgs=79 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:00.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 --2- 172.21.15.194:0/512606568 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff3f80f5410 0x7ff3f81093c0 unknown :-1 s=CLOSED pgs=226 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:00.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 >> 172.21.15.194:0/512606568 conn(0x7ff3f80f0b70 msgr2=0x7ff3f80f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:00.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.372+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 shutdown_connections 2024-01-27T23:29:00.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:00.373+0000 7ff400506700 1 -- 172.21.15.194:0/512606568 wait complete. 2024-01-27T23:29:00.365 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:00.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:28:59.291449+0000 mgr.smithi121.hcppvz (mgr.14162) 112 : cluster [DBG] pgmap v105: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:01.738 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:01.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:01.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:01 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:00.372274+0000 mon.smithi121 (mon.0) 192 : audit [DBG] from='client.? 172.21.15.194:0/512606568' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:02.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:01.291741+0000 mgr.smithi121.hcppvz (mgr.14162) 113 : cluster [DBG] pgmap v106: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:02.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.891+0000 7f80c1194700 1 -- 172.21.15.194:0/4079041087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 msgr2=0x7f80bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:02.883 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.891+0000 7f80c1194700 1 --2- 172.21.15.194:0/4079041087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc0f6930 secure :-1 s=READY pgs=227 cs=0 l=1 rev1=1 rx=0x7f80ac009fb0 tx=0x7f80ac009b30).stop 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.891+0000 7f80c1194700 1 -- 172.21.15.194:0/4079041087 shutdown_connections 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.891+0000 7f80c1194700 1 --2- 172.21.15.194:0/4079041087 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc0f6930 unknown :-1 s=CLOSED pgs=227 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.891+0000 7f80c1194700 1 -- 172.21.15.194:0/4079041087 >> 172.21.15.194:0/4079041087 conn(0x7f80bc0f1c70 msgr2=0x7f80bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.891+0000 7f80c1194700 1 -- 172.21.15.194:0/4079041087 shutdown_connections 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.891+0000 7f80c1194700 1 -- 172.21.15.194:0/4079041087 wait complete. 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.892+0000 7f80c1194700 1 Processor -- start 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.892+0000 7f80c1194700 1 -- start start 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.892+0000 7f80c1194700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:02.884 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.892+0000 7f80c1194700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80bc106c20 con 0x7f80bc0f6510 2024-01-27T23:29:02.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.892+0000 7f80bad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:02.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.892+0000 7f80bad9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:39890/0 (socket says 172.21.15.194:39890) 2024-01-27T23:29:02.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80bad9d700 1 -- 172.21.15.194:0/2039215918 learned_addr learned my addr 172.21.15.194:0/2039215918 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:02.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80bad9d700 1 -- 172.21.15.194:0/2039215918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ac005040 con 0x7f80bc0f6510 2024-01-27T23:29:02.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80bad9d700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f80ac014000 tx=0x7f80ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:02.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80a37fe700 1 -- 172.21.15.194:0/2039215918 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ac0095f0 con 0x7f80bc0f6510 2024-01-27T23:29:02.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80bc106e20 con 0x7f80bc0f6510 2024-01-27T23:29:02.886 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80a37fe700 1 -- 172.21.15.194:0/2039215918 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80ac007e70 con 0x7f80bc0f6510 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80a37fe700 1 -- 172.21.15.194:0/2039215918 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ac014ea0 con 0x7f80bc0f6510 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.893+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80bc1072b0 con 0x7f80bc0f6510 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.894+0000 7f80a37fe700 1 -- 172.21.15.194:0/2039215918 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80ac025070 con 0x7f80bc0f6510 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.894+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f809c004fa0 con 0x7f80bc0f6510 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.894+0000 7f80a37fe700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80a402ff60 0x7f80a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.894+0000 7f80a37fe700 1 -- 172.21.15.194:0/2039215918 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80ac040c50 con 0x7f80bc0f6510 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.894+0000 7f80ba59c700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80a402ff60 0x7f80a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:02.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.895+0000 7f80ba59c700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80a402ff60 0x7f80a4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f80b00096a0 tx=0x7f80b0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:02.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:02.899+0000 7f80a37fe700 1 -- 172.21.15.194:0/2039215918 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80ac026350 con 0x7f80bc0f6510 2024-01-27T23:29:03.097 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.106+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f809c005670 con 0x7f80bc0f6510 2024-01-27T23:29:03.098 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.106+0000 7f80a37fe700 1 -- 172.21.15.194:0/2039215918 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80ac025300 con 0x7f80bc0f6510 2024-01-27T23:29:03.098 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:03.098 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:03.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80a402ff60 msgr2=0x7f80a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:03.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80a402ff60 0x7f80a4032420 secure :-1 s=READY pgs=80 cs=0 l=1 rev1=1 rx=0x7f80b00096a0 tx=0x7f80b0006b40).stop 2024-01-27T23:29:03.099 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 msgr2=0x7f80bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:03.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc106730 secure :-1 s=READY pgs=228 cs=0 l=1 rev1=1 rx=0x7f80ac014000 tx=0x7f80ac00b040).stop 2024-01-27T23:29:03.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 shutdown_connections 2024-01-27T23:29:03.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80a402ff60 0x7f80a4032420 unknown :-1 s=CLOSED pgs=80 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:03.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 --2- 172.21.15.194:0/2039215918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80bc0f6510 0x7f80bc106730 unknown :-1 s=CLOSED pgs=228 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:03.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 >> 172.21.15.194:0/2039215918 conn(0x7f80bc0f1c70 msgr2=0x7f80bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:03.100 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 shutdown_connections 2024-01-27T23:29:03.101 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:03.108+0000 7f80c1194700 1 -- 172.21.15.194:0/2039215918 wait complete. 2024-01-27T23:29:03.101 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:03.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:03 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:03.107779+0000 mon.smithi121 (mon.0) 193 : audit [DBG] from='client.? 172.21.15.194:0/2039215918' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:04.506 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:04.507 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:04.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:03.292016+0000 mgr.smithi121.hcppvz (mgr.14162) 114 : cluster [DBG] pgmap v107: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 -- 172.21.15.194:0/542550329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 msgr2=0x7f3fbc0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 --2- 172.21.15.194:0/542550329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc0f55c0 secure :-1 s=READY pgs=229 cs=0 l=1 rev1=1 rx=0x7f3fb8009fb0 tx=0x7f3fb8009b30).stop 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 -- 172.21.15.194:0/542550329 shutdown_connections 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 --2- 172.21.15.194:0/542550329 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc0f55c0 unknown :-1 s=CLOSED pgs=229 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 -- 172.21.15.194:0/542550329 >> 172.21.15.194:0/542550329 conn(0x7f3fbc0f09a0 msgr2=0x7f3fbc0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 -- 172.21.15.194:0/542550329 shutdown_connections 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 -- 172.21.15.194:0/542550329 wait complete. 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 Processor -- start 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.677+0000 7f3fc2646700 1 -- start start 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc2646700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:05.670 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc2646700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3fbc060220 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc1644700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc1644700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40148/0 (socket says 172.21.15.194:40148) 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc1644700 1 -- 172.21.15.194:0/2455622608 learned_addr learned my addr 172.21.15.194:0/2455622608 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc1644700 1 -- 172.21.15.194:0/2455622608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3fb8005040 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc1644700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc1091d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3fb8013510 tx=0x7f3fb800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fb1ffb700 1 -- 172.21.15.194:0/2455622608 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb8009320 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fb1ffb700 1 -- 172.21.15.194:0/2455622608 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3fb8013f00 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3fbc060420 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.678+0000 7f3fb1ffb700 1 -- 172.21.15.194:0/2455622608 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3fb8014dc0 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.671 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.679+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3fbc109880 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.679+0000 7f3fb1ffb700 1 -- 172.21.15.194:0/2455622608 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3fb801d3d0 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.679+0000 7f3fb1ffb700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3fa802ff60 0x7f3fa8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:05.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.679+0000 7f3fb1ffb700 1 -- 172.21.15.194:0/2455622608 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3fb8041040 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.679+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3fbc059100 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.672 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.680+0000 7f3fc0e43700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3fa802ff60 0x7f3fa8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:05.673 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.680+0000 7f3fc0e43700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3fa802ff60 0x7f3fa8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3fac0096a0 tx=0x7f3fac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:05.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.684+0000 7f3fb1ffb700 1 -- 172.21.15.194:0/2455622608 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3fb8012920 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.894+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3fbc045ad0 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.885 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.894+0000 7f3fb1ffb700 1 -- 172.21.15.194:0/2455622608 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3fbc045ad0 con 0x7f3fbc0f51a0 2024-01-27T23:29:05.886 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:05.886 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:05.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3fa802ff60 msgr2=0x7f3fa8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:05.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3fa802ff60 0x7f3fa8032420 secure :-1 s=READY pgs=81 cs=0 l=1 rev1=1 rx=0x7f3fac0096a0 tx=0x7f3fac006b40).stop 2024-01-27T23:29:05.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 msgr2=0x7f3fbc1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:05.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc1091d0 secure :-1 s=READY pgs=230 cs=0 l=1 rev1=1 rx=0x7f3fb8013510 tx=0x7f3fb800b040).stop 2024-01-27T23:29:05.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 shutdown_connections 2024-01-27T23:29:05.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3fa802ff60 0x7f3fa8032420 unknown :-1 s=CLOSED pgs=81 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:05.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 --2- 172.21.15.194:0/2455622608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3fbc0f51a0 0x7f3fbc1091d0 unknown :-1 s=CLOSED pgs=230 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:05.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 >> 172.21.15.194:0/2455622608 conn(0x7f3fbc0f09a0 msgr2=0x7f3fbc0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:05.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 shutdown_connections 2024-01-27T23:29:05.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:05.896+0000 7f3fc2646700 1 -- 172.21.15.194:0/2455622608 wait complete. 2024-01-27T23:29:05.888 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:06.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:05.292382+0000 mgr.smithi121.hcppvz (mgr.14162) 115 : cluster [DBG] pgmap v108: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:06.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:05.895690+0000 mon.smithi121 (mon.0) 194 : audit [DBG] from='client.? 172.21.15.194:0/2455622608' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:07.275 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:07.275 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:08.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.443+0000 7f3cbc555700 1 -- 172.21.15.194:0/2409173530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 msgr2=0x7f3cb40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:08.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.443+0000 7f3cbc555700 1 --2- 172.21.15.194:0/2409173530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40f5830 secure :-1 s=READY pgs=231 cs=0 l=1 rev1=1 rx=0x7f3cb0009fb0 tx=0x7f3cb0009b30).stop 2024-01-27T23:29:08.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.443+0000 7f3cbc555700 1 -- 172.21.15.194:0/2409173530 shutdown_connections 2024-01-27T23:29:08.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.443+0000 7f3cbc555700 1 --2- 172.21.15.194:0/2409173530 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40f5830 unknown :-1 s=CLOSED pgs=231 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:08.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.443+0000 7f3cbc555700 1 -- 172.21.15.194:0/2409173530 >> 172.21.15.194:0/2409173530 conn(0x7f3cb40f0b70 msgr2=0x7f3cb40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:08.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.443+0000 7f3cbc555700 1 -- 172.21.15.194:0/2409173530 shutdown_connections 2024-01-27T23:29:08.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.443+0000 7f3cbc555700 1 -- 172.21.15.194:0/2409173530 wait complete. 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cbc555700 1 Processor -- start 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cbc555700 1 -- start start 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cbc555700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cbc555700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3cb40fb820 con 0x7f3cb40f5410 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cba2f1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cba2f1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40168/0 (socket says 172.21.15.194:40168) 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cba2f1700 1 -- 172.21.15.194:0/198813148 learned_addr learned my addr 172.21.15.194:0/198813148 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.444+0000 7f3cba2f1700 1 -- 172.21.15.194:0/198813148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3cb0005040 con 0x7f3cb40f5410 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.445+0000 7f3cba2f1700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40fd3a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3cb0013510 tx=0x7f3cb000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.445+0000 7f3caaffd700 1 -- 172.21.15.194:0/198813148 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cb0009320 con 0x7f3cb40f5410 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.445+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3cb40fb9c0 con 0x7f3cb40f5410 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.445+0000 7f3caaffd700 1 -- 172.21.15.194:0/198813148 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3cb0013f00 con 0x7f3cb40f5410 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.445+0000 7f3caaffd700 1 -- 172.21.15.194:0/198813148 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3cb0014d90 con 0x7f3cb40f5410 2024-01-27T23:29:08.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.445+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3cb40fbe50 con 0x7f3cb40f5410 2024-01-27T23:29:08.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.446+0000 7f3caaffd700 1 -- 172.21.15.194:0/198813148 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3cb0014420 con 0x7f3cb40f5410 2024-01-27T23:29:08.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.446+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3cb4059100 con 0x7f3cb40f5410 2024-01-27T23:29:08.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.446+0000 7f3caaffd700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ca002ff60 0x7f3ca0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:08.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.446+0000 7f3caaffd700 1 -- 172.21.15.194:0/198813148 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3cb0040ff0 con 0x7f3cb40f5410 2024-01-27T23:29:08.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.446+0000 7f3cb9af0700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ca002ff60 0x7f3ca0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:08.439 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.446+0000 7f3cb9af0700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ca002ff60 0x7f3ca0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3ca40096a0 tx=0x7f3ca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:08.441 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.450+0000 7f3caaffd700 1 -- 172.21.15.194:0/198813148 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3cb00095f0 con 0x7f3cb40f5410 2024-01-27T23:29:08.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.657+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3cb40fae60 con 0x7f3cb40f5410 2024-01-27T23:29:08.649 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.657+0000 7f3caaffd700 1 -- 172.21.15.194:0/198813148 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3cb001db70 con 0x7f3cb40f5410 2024-01-27T23:29:08.649 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:08.649 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:08.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ca002ff60 msgr2=0x7f3ca0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ca002ff60 0x7f3ca0032420 secure :-1 s=READY pgs=82 cs=0 l=1 rev1=1 rx=0x7f3ca40096a0 tx=0x7f3ca4006b40).stop 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 msgr2=0x7f3cb40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40fd3a0 secure :-1 s=READY pgs=232 cs=0 l=1 rev1=1 rx=0x7f3cb0013510 tx=0x7f3cb000b040).stop 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 shutdown_connections 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ca002ff60 0x7f3ca0032420 unknown :-1 s=CLOSED pgs=82 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 --2- 172.21.15.194:0/198813148 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3cb40f5410 0x7f3cb40fd3a0 unknown :-1 s=CLOSED pgs=232 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 >> 172.21.15.194:0/198813148 conn(0x7f3cb40f0b70 msgr2=0x7f3cb405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 shutdown_connections 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:08.659+0000 7f3cbc555700 1 -- 172.21.15.194:0/198813148 wait complete. 2024-01-27T23:29:08.651 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:08.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:07.292769+0000 mgr.smithi121.hcppvz (mgr.14162) 116 : cluster [DBG] pgmap v109: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:09.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:08.658907+0000 mon.smithi121 (mon.0) 195 : audit [DBG] from='client.? 172.21.15.194:0/198813148' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:10.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:10.032 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:10.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:09.293060+0000 mgr.smithi121.hcppvz (mgr.14162) 117 : cluster [DBG] pgmap v110: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:11.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.147+0000 7f30d0155700 1 -- 172.21.15.194:0/3847585033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 msgr2=0x7f30c80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:11.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.147+0000 7f30d0155700 1 --2- 172.21.15.194:0/3847585033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c80f6930 secure :-1 s=READY pgs=233 cs=0 l=1 rev1=1 rx=0x7f30b8009fb0 tx=0x7f30b8009b30).stop 2024-01-27T23:29:11.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.147+0000 7f30d0155700 1 -- 172.21.15.194:0/3847585033 shutdown_connections 2024-01-27T23:29:11.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.147+0000 7f30d0155700 1 --2- 172.21.15.194:0/3847585033 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c80f6930 unknown :-1 s=CLOSED pgs=233 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:11.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.147+0000 7f30d0155700 1 -- 172.21.15.194:0/3847585033 >> 172.21.15.194:0/3847585033 conn(0x7f30c80f1c70 msgr2=0x7f30c80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:11.139 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30d0155700 1 -- 172.21.15.194:0/3847585033 shutdown_connections 2024-01-27T23:29:11.140 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30d0155700 1 -- 172.21.15.194:0/3847585033 wait complete. 2024-01-27T23:29:11.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30d0155700 1 Processor -- start 2024-01-27T23:29:11.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30d0155700 1 -- start start 2024-01-27T23:29:11.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30d0155700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c8106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:11.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30d0155700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f30c8106c20 con 0x7f30c80f6510 2024-01-27T23:29:11.142 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30cdef1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c8106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30cdef1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c8106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40194/0 (socket says 172.21.15.194:40194) 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.148+0000 7f30cdef1700 1 -- 172.21.15.194:0/4196780143 learned_addr learned my addr 172.21.15.194:0/4196780143 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.149+0000 7f30cdef1700 1 -- 172.21.15.194:0/4196780143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f30b8005040 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.149+0000 7f30cdef1700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c8106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f30b8013510 tx=0x7f30b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.149+0000 7f30be7fc700 1 -- 172.21.15.194:0/4196780143 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30b8009320 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.149+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f30c8106e20 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.149+0000 7f30be7fc700 1 -- 172.21.15.194:0/4196780143 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f30b8013f00 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.149+0000 7f30be7fc700 1 -- 172.21.15.194:0/4196780143 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f30b8014d90 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.149+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f30c81072b0 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.150+0000 7f30be7fc700 1 -- 172.21.15.194:0/4196780143 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f30b80095f0 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.150+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f30c8045ad0 con 0x7f30c80f6510 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.150+0000 7f30be7fc700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f30b402ff60 0x7f30b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:11.143 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.150+0000 7f30be7fc700 1 -- 172.21.15.194:0/4196780143 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30b80410c0 con 0x7f30c80f6510 2024-01-27T23:29:11.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.150+0000 7f30cd6f0700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f30b402ff60 0x7f30b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:11.144 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.151+0000 7f30cd6f0700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f30b402ff60 0x7f30b4032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f30c40096a0 tx=0x7f30c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:11.145 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.154+0000 7f30be7fc700 1 -- 172.21.15.194:0/4196780143 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f30b8014420 con 0x7f30c80f6510 2024-01-27T23:29:11.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.361+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f30c80fbf60 con 0x7f30c80f6510 2024-01-27T23:29:11.353 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.362+0000 7f30be7fc700 1 -- 172.21.15.194:0/4196780143 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f30b8010230 con 0x7f30c80f6510 2024-01-27T23:29:11.354 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:11.354 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f30b402ff60 msgr2=0x7f30b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f30b402ff60 0x7f30b4032420 secure :-1 s=READY pgs=83 cs=0 l=1 rev1=1 rx=0x7f30c40096a0 tx=0x7f30c4006b40).stop 2024-01-27T23:29:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 msgr2=0x7f30c8106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c8106730 secure :-1 s=READY pgs=234 cs=0 l=1 rev1=1 rx=0x7f30b8013510 tx=0x7f30b800b040).stop 2024-01-27T23:29:11.355 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 shutdown_connections 2024-01-27T23:29:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f30b402ff60 0x7f30b4032420 unknown :-1 s=CLOSED pgs=83 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 --2- 172.21.15.194:0/4196780143 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f30c80f6510 0x7f30c8106730 unknown :-1 s=CLOSED pgs=234 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 >> 172.21.15.194:0/4196780143 conn(0x7f30c80f1c70 msgr2=0x7f30c80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 shutdown_connections 2024-01-27T23:29:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:11.364+0000 7f30d0155700 1 -- 172.21.15.194:0/4196780143 wait complete. 2024-01-27T23:29:11.356 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:12.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:11.293469+0000 mgr.smithi121.hcppvz (mgr.14162) 118 : cluster [DBG] pgmap v111: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:12.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:11.363336+0000 mon.smithi121 (mon.0) 196 : audit [DBG] from='client.? 172.21.15.194:0/4196780143' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:12.735 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:12.736 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.856+0000 7f022759e700 1 -- 172.21.15.194:0/1346796129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 msgr2=0x7f02280f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.856+0000 7f022759e700 1 --2- 172.21.15.194:0/1346796129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02280f55c0 secure :-1 s=READY pgs=235 cs=0 l=1 rev1=1 rx=0x7f021c009fb0 tx=0x7f021c009b30).stop 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.856+0000 7f022759e700 1 -- 172.21.15.194:0/1346796129 shutdown_connections 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.856+0000 7f022759e700 1 --2- 172.21.15.194:0/1346796129 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02280f55c0 unknown :-1 s=CLOSED pgs=235 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.856+0000 7f022759e700 1 -- 172.21.15.194:0/1346796129 >> 172.21.15.194:0/1346796129 conn(0x7f02280f09a0 msgr2=0x7f02280f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.856+0000 7f022759e700 1 -- 172.21.15.194:0/1346796129 shutdown_connections 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.856+0000 7f022759e700 1 -- 172.21.15.194:0/1346796129 wait complete. 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.857+0000 7f022759e700 1 Processor -- start 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.857+0000 7f022759e700 1 -- start start 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.857+0000 7f022759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02281053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:13.849 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.857+0000 7f022759e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f02281058a0 con 0x7f02280f51a0 2024-01-27T23:29:13.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.857+0000 7f022659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02281053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:13.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f022659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02281053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40202/0 (socket says 172.21.15.194:40202) 2024-01-27T23:29:13.850 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f022659c700 1 -- 172.21.15.194:0/1395840114 learned_addr learned my addr 172.21.15.194:0/1395840114 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:13.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f022659c700 1 -- 172.21.15.194:0/1395840114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f021c005040 con 0x7f02280f51a0 2024-01-27T23:29:13.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f022659c700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02281053b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f021c013510 tx=0x7f021c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:13.851 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f0216ffd700 1 -- 172.21.15.194:0/1395840114 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f021c009320 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0228105aa0 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f0216ffd700 1 -- 172.21.15.194:0/1395840114 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f021c013f00 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f0216ffd700 1 -- 172.21.15.194:0/1395840114 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f021c014dc0 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.858+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0228105f30 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.859+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0228045ad0 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.859+0000 7f0216ffd700 1 -- 172.21.15.194:0/1395840114 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f021c0095f0 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.859+0000 7f0216ffd700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f021002ff60 0x7f0210032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.859+0000 7f0216ffd700 1 -- 172.21.15.194:0/1395840114 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f021c041020 con 0x7f02280f51a0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.859+0000 7f0225d9b700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f021002ff60 0x7f0210032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:13.852 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.860+0000 7f0225d9b700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f021002ff60 0x7f0210032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f02180096a0 tx=0x7f0218006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:13.855 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:13.864+0000 7f0216ffd700 1 -- 172.21.15.194:0/1395840114 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f021c014420 con 0x7f02280f51a0 2024-01-27T23:29:14.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.073+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0228106d80 con 0x7f02280f51a0 2024-01-27T23:29:14.065 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.074+0000 7f0216ffd700 1 -- 172.21.15.194:0/1395840114 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f021c01c920 con 0x7f02280f51a0 2024-01-27T23:29:14.065 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:14.066 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:14.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.075+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f021002ff60 msgr2=0x7f0210032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:14.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.075+0000 7f022759e700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f021002ff60 0x7f0210032420 secure :-1 s=READY pgs=84 cs=0 l=1 rev1=1 rx=0x7f02180096a0 tx=0x7f0218006b40).stop 2024-01-27T23:29:14.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.075+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 msgr2=0x7f02281053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:14.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.075+0000 7f022759e700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02281053b0 secure :-1 s=READY pgs=236 cs=0 l=1 rev1=1 rx=0x7f021c013510 tx=0x7f021c00b040).stop 2024-01-27T23:29:14.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.076+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 shutdown_connections 2024-01-27T23:29:14.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.076+0000 7f022759e700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f021002ff60 0x7f0210032420 unknown :-1 s=CLOSED pgs=84 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:14.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.076+0000 7f022759e700 1 --2- 172.21.15.194:0/1395840114 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f02280f51a0 0x7f02281053b0 unknown :-1 s=CLOSED pgs=236 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:14.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.076+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 >> 172.21.15.194:0/1395840114 conn(0x7f02280f09a0 msgr2=0x7f02280f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:14.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.076+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 shutdown_connections 2024-01-27T23:29:14.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:14.076+0000 7f022759e700 1 -- 172.21.15.194:0/1395840114 wait complete. 2024-01-27T23:29:14.068 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:14.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:13.293941+0000 mgr.smithi121.hcppvz (mgr.14162) 119 : cluster [DBG] pgmap v112: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:14.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:14.075284+0000 mon.smithi121 (mon.0) 197 : audit [DBG] from='client.? 172.21.15.194:0/1395840114' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:15.389 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:15.390 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:16.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.542+0000 7f37fb95b700 1 -- 172.21.15.194:0/2777025793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 msgr2=0x7f37f40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:16.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.542+0000 7f37fb95b700 1 --2- 172.21.15.194:0/2777025793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f40f6930 secure :-1 s=READY pgs=237 cs=0 l=1 rev1=1 rx=0x7f37e8009fb0 tx=0x7f37e8009b30).stop 2024-01-27T23:29:16.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.542+0000 7f37fb95b700 1 -- 172.21.15.194:0/2777025793 shutdown_connections 2024-01-27T23:29:16.535 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.542+0000 7f37fb95b700 1 --2- 172.21.15.194:0/2777025793 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f40f6930 unknown :-1 s=CLOSED pgs=237 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.542+0000 7f37fb95b700 1 -- 172.21.15.194:0/2777025793 >> 172.21.15.194:0/2777025793 conn(0x7f37f40f1c70 msgr2=0x7f37f40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.542+0000 7f37fb95b700 1 -- 172.21.15.194:0/2777025793 shutdown_connections 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.542+0000 7f37fb95b700 1 -- 172.21.15.194:0/2777025793 wait complete. 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37fb95b700 1 Processor -- start 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37fb95b700 1 -- start start 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37fb95b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37fb95b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f37f4106c20 con 0x7f37f40f6510 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37f96f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37f96f7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45708/0 (socket says 172.21.15.194:45708) 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37f96f7700 1 -- 172.21.15.194:0/3237609554 learned_addr learned my addr 172.21.15.194:0/3237609554 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:16.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.543+0000 7f37f96f7700 1 -- 172.21.15.194:0/3237609554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f37e8005040 con 0x7f37f40f6510 2024-01-27T23:29:16.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.544+0000 7f37f96f7700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f4106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f37e8014000 tx=0x7f37e800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:16.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.544+0000 7f37e5ffb700 1 -- 172.21.15.194:0/3237609554 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e80095f0 con 0x7f37f40f6510 2024-01-27T23:29:16.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.544+0000 7f37e5ffb700 1 -- 172.21.15.194:0/3237609554 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f37e8007e70 con 0x7f37f40f6510 2024-01-27T23:29:16.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.544+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f37f4106e20 con 0x7f37f40f6510 2024-01-27T23:29:16.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.544+0000 7f37e5ffb700 1 -- 172.21.15.194:0/3237609554 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f37e8014ea0 con 0x7f37f40f6510 2024-01-27T23:29:16.537 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.544+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f37f41072b0 con 0x7f37f40f6510 2024-01-27T23:29:16.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.545+0000 7f37e5ffb700 1 -- 172.21.15.194:0/3237609554 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f37e8025070 con 0x7f37f40f6510 2024-01-27T23:29:16.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.545+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f37f4045ad0 con 0x7f37f40f6510 2024-01-27T23:29:16.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.545+0000 7f37e5ffb700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f37e002ff60 0x7f37e0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:16.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.545+0000 7f37e5ffb700 1 -- 172.21.15.194:0/3237609554 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f37e8040a30 con 0x7f37f40f6510 2024-01-27T23:29:16.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.545+0000 7f37f8ef6700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f37e002ff60 0x7f37e0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:16.538 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.546+0000 7f37f8ef6700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f37e002ff60 0x7f37e0032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f37f00096a0 tx=0x7f37f0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:16.541 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.550+0000 7f37e5ffb700 1 -- 172.21.15.194:0/3237609554 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f37e80079c0 con 0x7f37f40f6510 2024-01-27T23:29:16.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.759+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f37f40fbf60 con 0x7f37f40f6510 2024-01-27T23:29:16.751 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.760+0000 7f37e5ffb700 1 -- 172.21.15.194:0/3237609554 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f37e800f300 con 0x7f37f40f6510 2024-01-27T23:29:16.751 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:16.751 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:16.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f37e002ff60 msgr2=0x7f37e0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:16.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f37e002ff60 0x7f37e0032420 secure :-1 s=READY pgs=85 cs=0 l=1 rev1=1 rx=0x7f37f00096a0 tx=0x7f37f0006b40).stop 2024-01-27T23:29:16.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 msgr2=0x7f37f4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:16.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f4106730 secure :-1 s=READY pgs=238 cs=0 l=1 rev1=1 rx=0x7f37e8014000 tx=0x7f37e800b040).stop 2024-01-27T23:29:16.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 shutdown_connections 2024-01-27T23:29:16.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f37e002ff60 0x7f37e0032420 unknown :-1 s=CLOSED pgs=85 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:16.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 --2- 172.21.15.194:0/3237609554 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f37f40f6510 0x7f37f4106730 unknown :-1 s=CLOSED pgs=238 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:16.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 >> 172.21.15.194:0/3237609554 conn(0x7f37f40f1c70 msgr2=0x7f37f40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:16.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 shutdown_connections 2024-01-27T23:29:16.754 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:16.762+0000 7f37fb95b700 1 -- 172.21.15.194:0/3237609554 wait complete. 2024-01-27T23:29:16.754 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:16.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:15.294160+0000 mgr.smithi121.hcppvz (mgr.14162) 120 : cluster [DBG] pgmap v113: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:17.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:16.761324+0000 mon.smithi121 (mon.0) 198 : audit [DBG] from='client.? 172.21.15.194:0/3237609554' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:18.162 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:18.162 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:18.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:17.294381+0000 mgr.smithi121.hcppvz (mgr.14162) 121 : cluster [DBG] pgmap v114: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:18.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:17.395052+0000 mon.smithi121 (mon.0) 199 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:29:18.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:17.403673+0000 mon.smithi121 (mon.0) 200 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:29:19.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.290+0000 7fc71963c700 1 -- 172.21.15.194:0/685702906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 msgr2=0x7fc7140f57a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:19.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.290+0000 7fc71963c700 1 --2- 172.21.15.194:0/685702906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7140f57a0 secure :-1 s=READY pgs=239 cs=0 l=1 rev1=1 rx=0x7fc708009fb0 tx=0x7fc708009b30).stop 2024-01-27T23:29:19.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.290+0000 7fc71963c700 1 -- 172.21.15.194:0/685702906 shutdown_connections 2024-01-27T23:29:19.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.290+0000 7fc71963c700 1 --2- 172.21.15.194:0/685702906 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7140f57a0 unknown :-1 s=CLOSED pgs=239 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:19.282 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.290+0000 7fc71963c700 1 -- 172.21.15.194:0/685702906 >> 172.21.15.194:0/685702906 conn(0x7fc7140f0b00 msgr2=0x7fc7140f2f20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:19.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.290+0000 7fc71963c700 1 -- 172.21.15.194:0/685702906 shutdown_connections 2024-01-27T23:29:19.283 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.290+0000 7fc71963c700 1 -- 172.21.15.194:0/685702906 wait complete. 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.291+0000 7fc71963c700 1 Processor -- start 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.291+0000 7fc71963c700 1 -- start start 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.291+0000 7fc71963c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7141055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.291+0000 7fc71963c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc714105ac0 con 0x7fc7140f5380 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc712ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7141055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc712ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7141055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45728/0 (socket says 172.21.15.194:45728) 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc712ffd700 1 -- 172.21.15.194:0/3617583634 learned_addr learned my addr 172.21.15.194:0/3617583634 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc712ffd700 1 -- 172.21.15.194:0/3617583634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc708005040 con 0x7fc7140f5380 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc712ffd700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7141055d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fc708013510 tx=0x7fc70800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc6fb7fe700 1 -- 172.21.15.194:0/3617583634 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc708009320 con 0x7fc7140f5380 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc714105cc0 con 0x7fc7140f5380 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc714106150 con 0x7fc7140f5380 2024-01-27T23:29:19.285 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc6fb7fe700 1 -- 172.21.15.194:0/3617583634 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc708013f00 con 0x7fc7140f5380 2024-01-27T23:29:19.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.292+0000 7fc6fb7fe700 1 -- 172.21.15.194:0/3617583634 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc708014d60 con 0x7fc7140f5380 2024-01-27T23:29:19.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.293+0000 7fc6fb7fe700 1 -- 172.21.15.194:0/3617583634 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc7080263a0 con 0x7fc7140f5380 2024-01-27T23:29:19.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.293+0000 7fc6fb7fe700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc6fc02ff60 0x7fc6fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:19.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.293+0000 7fc6fb7fe700 1 -- 172.21.15.194:0/3617583634 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc708041360 con 0x7fc7140f5380 2024-01-27T23:29:19.286 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.293+0000 7fc7127fc700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc6fc02ff60 0x7fc6fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:19.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.294+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc714059100 con 0x7fc7140f5380 2024-01-27T23:29:19.287 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.294+0000 7fc7127fc700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc6fc02ff60 0x7fc6fc032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc7040096a0 tx=0x7fc704006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:19.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.298+0000 7fc6fb7fe700 1 -- 172.21.15.194:0/3617583634 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc70801d380 con 0x7fc7140f5380 2024-01-27T23:29:19.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.507+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc714045ad0 con 0x7fc7140f5380 2024-01-27T23:29:19.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.507+0000 7fc6fb7fe700 1 -- 172.21.15.194:0/3617583634 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc70802b070 con 0x7fc7140f5380 2024-01-27T23:29:19.499 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:19.499 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:19.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc6fc02ff60 msgr2=0x7fc6fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:19.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc6fc02ff60 0x7fc6fc032420 secure :-1 s=READY pgs=86 cs=0 l=1 rev1=1 rx=0x7fc7040096a0 tx=0x7fc704006b40).stop 2024-01-27T23:29:19.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 msgr2=0x7fc7141055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7141055d0 secure :-1 s=READY pgs=240 cs=0 l=1 rev1=1 rx=0x7fc708013510 tx=0x7fc70800b040).stop 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 shutdown_connections 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc6fc02ff60 0x7fc6fc032420 unknown :-1 s=CLOSED pgs=86 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 --2- 172.21.15.194:0/3617583634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc7140f5380 0x7fc7141055d0 unknown :-1 s=CLOSED pgs=240 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 >> 172.21.15.194:0/3617583634 conn(0x7fc7140f0b00 msgr2=0x7fc7140f15e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 shutdown_connections 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:19.509+0000 7fc71963c700 1 -- 172.21.15.194:0/3617583634 wait complete. 2024-01-27T23:29:19.501 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:20.838 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:19.294606+0000 mgr.smithi121.hcppvz (mgr.14162) 122 : cluster [DBG] pgmap v115: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:20.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:19.508711+0000 mon.smithi121 (mon.0) 201 : audit [DBG] from='client.? 172.21.15.194:0/3617583634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:20.873 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:20.873 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 -- 172.21.15.194:0/470828775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 msgr2=0x7f5f600f4b60 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 --2- 172.21.15.194:0/470828775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f600f4b60 secure :-1 s=READY pgs=241 cs=0 l=1 rev1=1 rx=0x7f5f50009fb0 tx=0x7f5f50009b30).stop 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 -- 172.21.15.194:0/470828775 shutdown_connections 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 --2- 172.21.15.194:0/470828775 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f600f4b60 unknown :-1 s=CLOSED pgs=241 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 -- 172.21.15.194:0/470828775 >> 172.21.15.194:0/470828775 conn(0x7f5f600f0260 msgr2=0x7f5f600f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 -- 172.21.15.194:0/470828775 shutdown_connections 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 -- 172.21.15.194:0/470828775 wait complete. 2024-01-27T23:29:22.079 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 Processor -- start 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 -- start start 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f601066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.086+0000 7f5f64c91700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5f60106ba0 con 0x7f5f600f4740 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f5e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f601066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f5e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f601066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45754/0 (socket says 172.21.15.194:45754) 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f5e59c700 1 -- 172.21.15.194:0/3600287658 learned_addr learned my addr 172.21.15.194:0/3600287658 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f5e59c700 1 -- 172.21.15.194:0/3600287658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5f50005040 con 0x7f5f600f4740 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f5e59c700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f601066b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5f50013510 tx=0x7f5f5000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f4effd700 1 -- 172.21.15.194:0/3600287658 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f50009320 con 0x7f5f600f4740 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f4effd700 1 -- 172.21.15.194:0/3600287658 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5f50013f00 con 0x7f5f600f4740 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5f60106da0 con 0x7f5f600f4740 2024-01-27T23:29:22.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f4effd700 1 -- 172.21.15.194:0/3600287658 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5f50014d90 con 0x7f5f600f4740 2024-01-27T23:29:22.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.087+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5f60107230 con 0x7f5f600f4740 2024-01-27T23:29:22.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.088+0000 7f5f4effd700 1 -- 172.21.15.194:0/3600287658 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5f50014420 con 0x7f5f600f4740 2024-01-27T23:29:22.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.088+0000 7f5f4effd700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5f4802ff60 0x7f5f48032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:22.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.088+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5f60045ad0 con 0x7f5f600f4740 2024-01-27T23:29:22.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.088+0000 7f5f4effd700 1 -- 172.21.15.194:0/3600287658 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5f50041000 con 0x7f5f600f4740 2024-01-27T23:29:22.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.089+0000 7f5f5dd9b700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5f4802ff60 0x7f5f48032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:22.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.089+0000 7f5f5dd9b700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5f4802ff60 0x7f5f48032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5f540096a0 tx=0x7f5f54006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:22.084 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.093+0000 7f5f4effd700 1 -- 172.21.15.194:0/3600287658 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5f5006d2b0 con 0x7f5f600f4740 2024-01-27T23:29:22.290 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.299+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5f601080e0 con 0x7f5f600f4740 2024-01-27T23:29:22.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.299+0000 7f5f4effd700 1 -- 172.21.15.194:0/3600287658 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5f5001dd50 con 0x7f5f600f4740 2024-01-27T23:29:22.340 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:22.340 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5f4802ff60 msgr2=0x7f5f48032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5f4802ff60 0x7f5f48032420 secure :-1 s=READY pgs=87 cs=0 l=1 rev1=1 rx=0x7f5f540096a0 tx=0x7f5f54006b40).stop 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 msgr2=0x7f5f601066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f601066b0 secure :-1 s=READY pgs=242 cs=0 l=1 rev1=1 rx=0x7f5f50013510 tx=0x7f5f5000b040).stop 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 shutdown_connections 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5f4802ff60 0x7f5f48032420 unknown :-1 s=CLOSED pgs=87 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 --2- 172.21.15.194:0/3600287658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5f600f4740 0x7f5f601066b0 unknown :-1 s=CLOSED pgs=242 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 >> 172.21.15.194:0/3600287658 conn(0x7f5f600f0260 msgr2=0x7f5f600f0d90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 shutdown_connections 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:22.301+0000 7f5f64c91700 1 -- 172.21.15.194:0/3600287658 wait complete. 2024-01-27T23:29:22.341 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:22.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:21.294938+0000 mgr.smithi121.hcppvz (mgr.14162) 123 : cluster [DBG] pgmap v116: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:23.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:23 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:22.300732+0000 mon.smithi121 (mon.0) 202 : audit [DBG] from='client.? 172.21.15.194:0/3600287658' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:23.674 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:23.675 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:24.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:23.295353+0000 mgr.smithi121.hcppvz (mgr.14162) 124 : cluster [DBG] pgmap v117: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:24.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.901+0000 7f34dabe1700 1 -- 172.21.15.194:0/3313627094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 msgr2=0x7f34d40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:24.897 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.901+0000 7f34dabe1700 1 --2- 172.21.15.194:0/3313627094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d40f6930 secure :-1 s=READY pgs=243 cs=0 l=1 rev1=1 rx=0x7f34c8009fb0 tx=0x7f34c8009b30).stop 2024-01-27T23:29:24.897 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.901+0000 7f34dabe1700 1 -- 172.21.15.194:0/3313627094 shutdown_connections 2024-01-27T23:29:24.897 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.901+0000 7f34dabe1700 1 --2- 172.21.15.194:0/3313627094 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d40f6930 unknown :-1 s=CLOSED pgs=243 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:24.897 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.901+0000 7f34dabe1700 1 -- 172.21.15.194:0/3313627094 >> 172.21.15.194:0/3313627094 conn(0x7f34d40f1c70 msgr2=0x7f34d40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:24.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.901+0000 7f34dabe1700 1 -- 172.21.15.194:0/3313627094 shutdown_connections 2024-01-27T23:29:24.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.901+0000 7f34dabe1700 1 -- 172.21.15.194:0/3313627094 wait complete. 2024-01-27T23:29:24.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34dabe1700 1 Processor -- start 2024-01-27T23:29:24.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34dabe1700 1 -- start start 2024-01-27T23:29:24.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34dabe1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d4106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:24.898 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34dabe1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f34d4106c20 con 0x7f34d40f6510 2024-01-27T23:29:24.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34d897d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d4106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:24.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34d897d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d4106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45770/0 (socket says 172.21.15.194:45770) 2024-01-27T23:29:24.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34d897d700 1 -- 172.21.15.194:0/4238360424 learned_addr learned my addr 172.21.15.194:0/4238360424 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:24.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.902+0000 7f34d897d700 1 -- 172.21.15.194:0/4238360424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f34c8005040 con 0x7f34d40f6510 2024-01-27T23:29:24.899 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.903+0000 7f34d897d700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d4106730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f34c8013510 tx=0x7f34c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:24.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.903+0000 7f34d17fa700 1 -- 172.21.15.194:0/4238360424 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c8009320 con 0x7f34d40f6510 2024-01-27T23:29:24.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.903+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f34d4106e20 con 0x7f34d40f6510 2024-01-27T23:29:24.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.903+0000 7f34d17fa700 1 -- 172.21.15.194:0/4238360424 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f34c8013f00 con 0x7f34d40f6510 2024-01-27T23:29:24.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.903+0000 7f34d17fa700 1 -- 172.21.15.194:0/4238360424 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f34c8014d90 con 0x7f34d40f6510 2024-01-27T23:29:24.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.903+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f34d41072b0 con 0x7f34d40f6510 2024-01-27T23:29:24.900 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.904+0000 7f34d17fa700 1 -- 172.21.15.194:0/4238360424 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f34c8014420 con 0x7f34d40f6510 2024-01-27T23:29:24.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.904+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f34d4045ad0 con 0x7f34d40f6510 2024-01-27T23:29:24.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.904+0000 7f34d17fa700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f34c402ff60 0x7f34c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:24.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.904+0000 7f34d17fa700 1 -- 172.21.15.194:0/4238360424 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f34c8041110 con 0x7f34d40f6510 2024-01-27T23:29:24.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.904+0000 7f34d3fff700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f34c402ff60 0x7f34c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:24.901 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.904+0000 7f34d3fff700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f34c402ff60 0x7f34c4032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f34c00096a0 tx=0x7f34c0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:24.902 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:24.908+0000 7f34d17fa700 1 -- 172.21.15.194:0/4238360424 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34c80095f0 con 0x7f34d40f6510 2024-01-27T23:29:25.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.112+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f34d40fbf60 con 0x7f34d40f6510 2024-01-27T23:29:25.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.113+0000 7f34d17fa700 1 -- 172.21.15.194:0/4238360424 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f34c801db70 con 0x7f34d40f6510 2024-01-27T23:29:25.104 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:25.105 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:25.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.114+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f34c402ff60 msgr2=0x7f34c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:25.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.114+0000 7f34dabe1700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f34c402ff60 0x7f34c4032420 secure :-1 s=READY pgs=88 cs=0 l=1 rev1=1 rx=0x7f34c00096a0 tx=0x7f34c0006b40).stop 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.114+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 msgr2=0x7f34d4106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.114+0000 7f34dabe1700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d4106730 secure :-1 s=READY pgs=244 cs=0 l=1 rev1=1 rx=0x7f34c8013510 tx=0x7f34c800b040).stop 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.115+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 shutdown_connections 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.115+0000 7f34dabe1700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f34c402ff60 0x7f34c4032420 unknown :-1 s=CLOSED pgs=88 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.115+0000 7f34dabe1700 1 --2- 172.21.15.194:0/4238360424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f34d40f6510 0x7f34d4106730 unknown :-1 s=CLOSED pgs=244 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.115+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 >> 172.21.15.194:0/4238360424 conn(0x7f34d40f1c70 msgr2=0x7f34d40f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.115+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 shutdown_connections 2024-01-27T23:29:25.107 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:25.115+0000 7f34dabe1700 1 -- 172.21.15.194:0/4238360424 wait complete. 2024-01-27T23:29:25.108 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:25.588 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:25.114262+0000 mon.smithi121 (mon.0) 203 : audit [DBG] from='client.? 172.21.15.194:0/4238360424' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:26.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:26.531 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:25.295641+0000 mgr.smithi121.hcppvz (mgr.14162) 125 : cluster [DBG] pgmap v118: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:27.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.712+0000 7fb8a691e700 1 -- 172.21.15.194:0/253248691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 msgr2=0x7fb8a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:27.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.712+0000 7fb8a691e700 1 --2- 172.21.15.194:0/253248691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a00f5830 secure :-1 s=READY pgs=245 cs=0 l=1 rev1=1 rx=0x7fb890009fb0 tx=0x7fb890009b30).stop 2024-01-27T23:29:27.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.712+0000 7fb8a691e700 1 -- 172.21.15.194:0/253248691 shutdown_connections 2024-01-27T23:29:27.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.712+0000 7fb8a691e700 1 --2- 172.21.15.194:0/253248691 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a00f5830 unknown :-1 s=CLOSED pgs=245 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:27.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.712+0000 7fb8a691e700 1 -- 172.21.15.194:0/253248691 >> 172.21.15.194:0/253248691 conn(0x7fb8a00f0b70 msgr2=0x7fb8a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:27.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.712+0000 7fb8a691e700 1 -- 172.21.15.194:0/253248691 shutdown_connections 2024-01-27T23:29:27.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.712+0000 7fb8a691e700 1 -- 172.21.15.194:0/253248691 wait complete. 2024-01-27T23:29:27.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.713+0000 7fb8a691e700 1 Processor -- start 2024-01-27T23:29:27.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.713+0000 7fb8a691e700 1 -- start start 2024-01-27T23:29:27.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.713+0000 7fb8a691e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:27.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.713+0000 7fb8a691e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8a0105b20 con 0x7fb8a00f5410 2024-01-27T23:29:27.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:27.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55498/0 (socket says 172.21.15.194:55498) 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89ffff700 1 -- 172.21.15.194:0/386656375 learned_addr learned my addr 172.21.15.194:0/386656375 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89ffff700 1 -- 172.21.15.194:0/386656375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb890005040 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89ffff700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a0105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb890013510 tx=0x7fb89000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89cff9700 1 -- 172.21.15.194:0/386656375 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb890009320 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8a0105d20 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89cff9700 1 -- 172.21.15.194:0/386656375 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb890013f00 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8a01061b0 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.714+0000 7fb89cff9700 1 -- 172.21.15.194:0/386656375 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb890014d90 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.715+0000 7fb89cff9700 1 -- 172.21.15.194:0/386656375 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb89000f410 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.715+0000 7fb89cff9700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb88802ffb0 0x7fb888032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.715+0000 7fb89cff9700 1 -- 172.21.15.194:0/386656375 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb890042e60 con 0x7fb8a00f5410 2024-01-27T23:29:27.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.715+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8a0045ad0 con 0x7fb8a00f5410 2024-01-27T23:29:27.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.715+0000 7fb89f7fe700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb88802ffb0 0x7fb888032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:27.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.716+0000 7fb89f7fe700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb88802ffb0 0x7fb888032470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fb8940096a0 tx=0x7fb894006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:27.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.720+0000 7fb89cff9700 1 -- 172.21.15.194:0/386656375 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb890012920 con 0x7fb8a00f5410 2024-01-27T23:29:27.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.926+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8a00faf50 con 0x7fb8a00f5410 2024-01-27T23:29:27.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.927+0000 7fb89cff9700 1 -- 172.21.15.194:0/386656375 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb890012330 con 0x7fb8a00f5410 2024-01-27T23:29:27.919 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:27.919 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:27.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.928+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb88802ffb0 msgr2=0x7fb888032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:27.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.928+0000 7fb8a691e700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb88802ffb0 0x7fb888032470 secure :-1 s=READY pgs=89 cs=0 l=1 rev1=1 rx=0x7fb8940096a0 tx=0x7fb894006b40).stop 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.928+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 msgr2=0x7fb8a0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.928+0000 7fb8a691e700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a0105630 secure :-1 s=READY pgs=246 cs=0 l=1 rev1=1 rx=0x7fb890013510 tx=0x7fb89000b040).stop 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.929+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 shutdown_connections 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.929+0000 7fb8a691e700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb88802ffb0 0x7fb888032470 unknown :-1 s=CLOSED pgs=89 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.929+0000 7fb8a691e700 1 --2- 172.21.15.194:0/386656375 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8a00f5410 0x7fb8a0105630 unknown :-1 s=CLOSED pgs=246 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.929+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 >> 172.21.15.194:0/386656375 conn(0x7fb8a00f0b70 msgr2=0x7fb8a00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.929+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 shutdown_connections 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:27.929+0000 7fb8a691e700 1 -- 172.21.15.194:0/386656375 wait complete. 2024-01-27T23:29:27.921 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:28.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:27.295948+0000 mgr.smithi121.hcppvz (mgr.14162) 126 : cluster [DBG] pgmap v119: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:28.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:27.927952+0000 mon.smithi121 (mon.0) 204 : audit [DBG] from='client.? 172.21.15.194:0/386656375' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:29.337 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:29.337 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:30.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.419+0000 7f15d92ef700 1 -- 172.21.15.194:0/252724154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 msgr2=0x7f15d40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:30.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.419+0000 7f15d92ef700 1 --2- 172.21.15.194:0/252724154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d40f5830 secure :-1 s=READY pgs=247 cs=0 l=1 rev1=1 rx=0x7f15c8009fb0 tx=0x7f15c8009b30).stop 2024-01-27T23:29:30.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.419+0000 7f15d92ef700 1 -- 172.21.15.194:0/252724154 shutdown_connections 2024-01-27T23:29:30.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.419+0000 7f15d92ef700 1 --2- 172.21.15.194:0/252724154 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d40f5830 unknown :-1 s=CLOSED pgs=247 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:30.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.419+0000 7f15d92ef700 1 -- 172.21.15.194:0/252724154 >> 172.21.15.194:0/252724154 conn(0x7f15d40f0b70 msgr2=0x7f15d40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:30.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.419+0000 7f15d92ef700 1 -- 172.21.15.194:0/252724154 shutdown_connections 2024-01-27T23:29:30.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.419+0000 7f15d92ef700 1 -- 172.21.15.194:0/252724154 wait complete. 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.420+0000 7f15d92ef700 1 Processor -- start 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.420+0000 7f15d92ef700 1 -- start start 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.420+0000 7f15d92ef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.420+0000 7f15d92ef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f15d4105b20 con 0x7f15d40f5410 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15d2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15d2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55524/0 (socket says 172.21.15.194:55524) 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15d2ffd700 1 -- 172.21.15.194:0/2740200450 learned_addr learned my addr 172.21.15.194:0/2740200450 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15d2ffd700 1 -- 172.21.15.194:0/2740200450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f15c8005040 con 0x7f15d40f5410 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15d2ffd700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d4105630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f15c8014000 tx=0x7f15c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15bb7fe700 1 -- 172.21.15.194:0/2740200450 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15c80095f0 con 0x7f15d40f5410 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f15d4105d20 con 0x7f15d40f5410 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15bb7fe700 1 -- 172.21.15.194:0/2740200450 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f15c8007e70 con 0x7f15d40f5410 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15bb7fe700 1 -- 172.21.15.194:0/2740200450 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f15c8014ea0 con 0x7f15d40f5410 2024-01-27T23:29:30.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.421+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f15d41061b0 con 0x7f15d40f5410 2024-01-27T23:29:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.422+0000 7f15bb7fe700 1 -- 172.21.15.194:0/2740200450 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f15c8027070 con 0x7f15d40f5410 2024-01-27T23:29:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.422+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f15d4045ad0 con 0x7f15d40f5410 2024-01-27T23:29:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.422+0000 7f15bb7fe700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15bc02ffb0 0x7f15bc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.422+0000 7f15bb7fe700 1 -- 172.21.15.194:0/2740200450 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f15c80427d0 con 0x7f15d40f5410 2024-01-27T23:29:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.422+0000 7f15d27fc700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15bc02ffb0 0x7f15bc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:30.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.423+0000 7f15d27fc700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15bc02ffb0 0x7f15bc032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f15c40096a0 tx=0x7f15c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:30.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.427+0000 7f15bb7fe700 1 -- 172.21.15.194:0/2740200450 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f15c80274e0 con 0x7f15d40f5410 2024-01-27T23:29:30.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:29.296288+0000 mgr.smithi121.hcppvz (mgr.14162) 127 : cluster [DBG] pgmap v120: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:30.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.634+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f15d40fae60 con 0x7f15d40f5410 2024-01-27T23:29:30.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.635+0000 7f15bb7fe700 1 -- 172.21.15.194:0/2740200450 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f15c8027300 con 0x7f15d40f5410 2024-01-27T23:29:30.627 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:30.627 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15bc02ffb0 msgr2=0x7f15bc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15bc02ffb0 0x7f15bc032470 secure :-1 s=READY pgs=90 cs=0 l=1 rev1=1 rx=0x7f15c40096a0 tx=0x7f15c4006b40).stop 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 msgr2=0x7f15d4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d4105630 secure :-1 s=READY pgs=248 cs=0 l=1 rev1=1 rx=0x7f15c8014000 tx=0x7f15c800b040).stop 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 shutdown_connections 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f15bc02ffb0 0x7f15bc032470 unknown :-1 s=CLOSED pgs=90 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 --2- 172.21.15.194:0/2740200450 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f15d40f5410 0x7f15d4105630 unknown :-1 s=CLOSED pgs=248 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 >> 172.21.15.194:0/2740200450 conn(0x7f15d40f0b70 msgr2=0x7f15d40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 shutdown_connections 2024-01-27T23:29:30.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:30.637+0000 7f15d92ef700 1 -- 172.21.15.194:0/2740200450 wait complete. 2024-01-27T23:29:30.630 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:31.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:31 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:30.636509+0000 mon.smithi121 (mon.0) 205 : audit [DBG] from='client.? 172.21.15.194:0/2740200450' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:32.001 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:32.002 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:32.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:31.296538+0000 mgr.smithi121.hcppvz (mgr.14162) 128 : cluster [DBG] pgmap v121: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.196+0000 7f3af3560700 1 -- 172.21.15.194:0/3945243356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 msgr2=0x7f3aec0f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.196+0000 7f3af3560700 1 --2- 172.21.15.194:0/3945243356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec0f9190 secure :-1 s=READY pgs=249 cs=0 l=1 rev1=1 rx=0x7f3ae0009fb0 tx=0x7f3ae0009b30).stop 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.196+0000 7f3af3560700 1 -- 172.21.15.194:0/3945243356 shutdown_connections 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.196+0000 7f3af3560700 1 --2- 172.21.15.194:0/3945243356 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec0f9190 unknown :-1 s=CLOSED pgs=249 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.196+0000 7f3af3560700 1 -- 172.21.15.194:0/3945243356 >> 172.21.15.194:0/3945243356 conn(0x7f3aec0f0b70 msgr2=0x7f3aec0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.196+0000 7f3af3560700 1 -- 172.21.15.194:0/3945243356 shutdown_connections 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.196+0000 7f3af3560700 1 -- 172.21.15.194:0/3945243356 wait complete. 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af3560700 1 Processor -- start 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af3560700 1 -- start start 2024-01-27T23:29:33.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af3560700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:33.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af3560700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3aec106bd0 con 0x7f3aec0601e0 2024-01-27T23:29:33.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af12fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:33.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af12fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55540/0 (socket says 172.21.15.194:55540) 2024-01-27T23:29:33.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af12fc700 1 -- 172.21.15.194:0/4220803072 learned_addr learned my addr 172.21.15.194:0/4220803072 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:33.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.197+0000 7f3af12fc700 1 -- 172.21.15.194:0/4220803072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3ae0005040 con 0x7f3aec0601e0 2024-01-27T23:29:33.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.198+0000 7f3af12fc700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec1066e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f3ae0004dd0 tx=0x7f3ae0009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.198+0000 7f3addffb700 1 -- 172.21.15.194:0/4220803072 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae0018070 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.198+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3aec106dd0 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.198+0000 7f3addffb700 1 -- 172.21.15.194:0/4220803072 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3ae0013410 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.198+0000 7f3addffb700 1 -- 172.21.15.194:0/4220803072 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3ae001c3c0 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.198+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3aec107260 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.199+0000 7f3addffb700 1 -- 172.21.15.194:0/4220803072 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3ae001c630 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.199+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3aec045ad0 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.199+0000 7f3addffb700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ad802ff60 0x7f3ad8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.199+0000 7f3addffb700 1 -- 172.21.15.194:0/4220803072 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3ae0041720 con 0x7f3aec0601e0 2024-01-27T23:29:33.192 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.199+0000 7f3af0afb700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ad802ff60 0x7f3ad8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:33.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.199+0000 7f3af0afb700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ad802ff60 0x7f3ad8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3ae80096a0 tx=0x7f3ae8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:33.195 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.204+0000 7f3addffb700 1 -- 172.21.15.194:0/4220803072 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3ae001ca30 con 0x7f3aec0601e0 2024-01-27T23:29:33.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.410+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3aec1080b0 con 0x7f3aec0601e0 2024-01-27T23:29:33.402 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.411+0000 7f3addffb700 1 -- 172.21.15.194:0/4220803072 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3ae001b9c0 con 0x7f3aec0601e0 2024-01-27T23:29:33.402 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:33.403 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:33.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ad802ff60 msgr2=0x7f3ad8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:33.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ad802ff60 0x7f3ad8032420 secure :-1 s=READY pgs=91 cs=0 l=1 rev1=1 rx=0x7f3ae80096a0 tx=0x7f3ae8006b40).stop 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 msgr2=0x7f3aec1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec1066e0 secure :-1 s=READY pgs=250 cs=0 l=1 rev1=1 rx=0x7f3ae0004dd0 tx=0x7f3ae0009b30).stop 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 shutdown_connections 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3ad802ff60 0x7f3ad8032420 unknown :-1 s=CLOSED pgs=91 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 --2- 172.21.15.194:0/4220803072 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3aec0601e0 0x7f3aec1066e0 unknown :-1 s=CLOSED pgs=250 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 >> 172.21.15.194:0/4220803072 conn(0x7f3aec0f0b70 msgr2=0x7f3aec0f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 shutdown_connections 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:33.412+0000 7f3af3560700 1 -- 172.21.15.194:0/4220803072 wait complete. 2024-01-27T23:29:33.405 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:34.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:33.296892+0000 mgr.smithi121.hcppvz (mgr.14162) 129 : cluster [DBG] pgmap v122: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:34.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:33.411883+0000 mon.smithi121 (mon.0) 206 : audit [DBG] from='client.? 172.21.15.194:0/4220803072' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:34.826 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:34.827 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:35.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 -- 172.21.15.194:0/2351432175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 msgr2=0x7f24a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:35.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 --2- 172.21.15.194:0/2351432175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a40f5830 secure :-1 s=READY pgs=251 cs=0 l=1 rev1=1 rx=0x7f2498009fb0 tx=0x7f2498009b30).stop 2024-01-27T23:29:35.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 -- 172.21.15.194:0/2351432175 shutdown_connections 2024-01-27T23:29:35.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 --2- 172.21.15.194:0/2351432175 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a40f5830 unknown :-1 s=CLOSED pgs=251 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:35.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 -- 172.21.15.194:0/2351432175 >> 172.21.15.194:0/2351432175 conn(0x7f24a40f0b70 msgr2=0x7f24a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:35.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 -- 172.21.15.194:0/2351432175 shutdown_connections 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 -- 172.21.15.194:0/2351432175 wait complete. 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 Processor -- start 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 -- start start 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a4109420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.994+0000 7f24a95cc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f24a4060220 con 0x7f24a40f5410 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f24a2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a4109420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f24a2ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a4109420 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37498/0 (socket says 172.21.15.194:37498) 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f24a2ffd700 1 -- 172.21.15.194:0/3553203973 learned_addr learned my addr 172.21.15.194:0/3553203973 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f24a2ffd700 1 -- 172.21.15.194:0/3553203973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2498005040 con 0x7f24a40f5410 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f24a2ffd700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a4109420 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f2498013510 tx=0x7f249800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f248b7fe700 1 -- 172.21.15.194:0/3553203973 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2498009320 con 0x7f24a40f5410 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f248b7fe700 1 -- 172.21.15.194:0/3553203973 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2498013f00 con 0x7f24a40f5410 2024-01-27T23:29:35.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f24a4060420 con 0x7f24a40f5410 2024-01-27T23:29:35.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f248b7fe700 1 -- 172.21.15.194:0/3553203973 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2498014d90 con 0x7f24a40f5410 2024-01-27T23:29:35.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.995+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24a4109ad0 con 0x7f24a40f5410 2024-01-27T23:29:35.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.996+0000 7f248b7fe700 1 -- 172.21.15.194:0/3553203973 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2498014420 con 0x7f24a40f5410 2024-01-27T23:29:35.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.996+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f24a4059100 con 0x7f24a40f5410 2024-01-27T23:29:35.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.996+0000 7f248b7fe700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f248c02ffb0 0x7f248c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:35.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.996+0000 7f248b7fe700 1 -- 172.21.15.194:0/3553203973 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2498041090 con 0x7f24a40f5410 2024-01-27T23:29:35.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.996+0000 7f24a27fc700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f248c02ffb0 0x7f248c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:35.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:35.997+0000 7f24a27fc700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f248c02ffb0 0x7f248c032470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f24940096a0 tx=0x7f2494006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:35.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.001+0000 7f248b7fe700 1 -- 172.21.15.194:0/3553203973 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f249806d2b0 con 0x7f24a40f5410 2024-01-27T23:29:36.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.209+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24a4045ad0 con 0x7f24a40f5410 2024-01-27T23:29:36.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.210+0000 7f248b7fe700 1 -- 172.21.15.194:0/3553203973 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f24980095f0 con 0x7f24a40f5410 2024-01-27T23:29:36.202 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:36.202 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:36.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f248c02ffb0 msgr2=0x7f248c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:36.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f248c02ffb0 0x7f248c032470 secure :-1 s=READY pgs=92 cs=0 l=1 rev1=1 rx=0x7f24940096a0 tx=0x7f2494006b40).stop 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 msgr2=0x7f24a4109420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a4109420 secure :-1 s=READY pgs=252 cs=0 l=1 rev1=1 rx=0x7f2498013510 tx=0x7f249800b040).stop 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 shutdown_connections 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f248c02ffb0 0x7f248c032470 unknown :-1 s=CLOSED pgs=92 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 --2- 172.21.15.194:0/3553203973 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24a40f5410 0x7f24a4109420 unknown :-1 s=CLOSED pgs=252 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 >> 172.21.15.194:0/3553203973 conn(0x7f24a40f0b70 msgr2=0x7f24a40f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 shutdown_connections 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:36.212+0000 7f24a95cc700 1 -- 172.21.15.194:0/3553203973 wait complete. 2024-01-27T23:29:36.204 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:36.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:35.297273+0000 mgr.smithi121.hcppvz (mgr.14162) 130 : cluster [DBG] pgmap v123: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:36.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:36.211436+0000 mon.smithi121 (mon.0) 207 : audit [DBG] from='client.? 172.21.15.194:0/3553203973' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:37.596 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:37.596 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:38.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:37.297522+0000 mgr.smithi121.hcppvz (mgr.14162) 131 : cluster [DBG] pgmap v124: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:38.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.781+0000 7f94968d7700 1 -- 172.21.15.194:0/1075564723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 msgr2=0x7f94900fb550 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:38.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.781+0000 7f94968d7700 1 --2- 172.21.15.194:0/1075564723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94900fb550 secure :-1 s=READY pgs=253 cs=0 l=1 rev1=1 rx=0x7f9480009fb0 tx=0x7f9480009b30).stop 2024-01-27T23:29:38.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.781+0000 7f94968d7700 1 -- 172.21.15.194:0/1075564723 shutdown_connections 2024-01-27T23:29:38.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.781+0000 7f94968d7700 1 --2- 172.21.15.194:0/1075564723 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94900fb550 unknown :-1 s=CLOSED pgs=253 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:38.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.781+0000 7f94968d7700 1 -- 172.21.15.194:0/1075564723 >> 172.21.15.194:0/1075564723 conn(0x7f94900f0b70 msgr2=0x7f94900f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:38.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.781+0000 7f94968d7700 1 -- 172.21.15.194:0/1075564723 shutdown_connections 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.781+0000 7f94968d7700 1 -- 172.21.15.194:0/1075564723 wait complete. 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.782+0000 7f94968d7700 1 Processor -- start 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.782+0000 7f94968d7700 1 -- start start 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.782+0000 7f94968d7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94901088b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.782+0000 7f94968d7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9490108da0 con 0x7f9490060220 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.782+0000 7f948ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94901088b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.782+0000 7f948ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94901088b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37516/0 (socket says 172.21.15.194:37516) 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.782+0000 7f948ffff700 1 -- 172.21.15.194:0/3763181886 learned_addr learned my addr 172.21.15.194:0/3763181886 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.783+0000 7f948ffff700 1 -- 172.21.15.194:0/3763181886 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9480005040 con 0x7f9490060220 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.783+0000 7f948ffff700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94901088b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9480013510 tx=0x7f948000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.783+0000 7f948cff9700 1 -- 172.21.15.194:0/3763181886 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9480009320 con 0x7f9490060220 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.783+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9490108fa0 con 0x7f9490060220 2024-01-27T23:29:38.776 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.783+0000 7f948cff9700 1 -- 172.21.15.194:0/3763181886 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9480013f00 con 0x7f9490060220 2024-01-27T23:29:38.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.783+0000 7f948cff9700 1 -- 172.21.15.194:0/3763181886 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9480014dc0 con 0x7f9490060220 2024-01-27T23:29:38.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.783+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9490109430 con 0x7f9490060220 2024-01-27T23:29:38.777 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.784+0000 7f948cff9700 1 -- 172.21.15.194:0/3763181886 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f948001d3d0 con 0x7f9490060220 2024-01-27T23:29:38.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.784+0000 7f948cff9700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f947c02ff60 0x7f947c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:38.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.784+0000 7f948cff9700 1 -- 172.21.15.194:0/3763181886 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f94800410b0 con 0x7f9490060220 2024-01-27T23:29:38.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.784+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9490045ad0 con 0x7f9490060220 2024-01-27T23:29:38.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.784+0000 7f948f7fe700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f947c02ff60 0x7f947c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:38.778 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.785+0000 7f948f7fe700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f947c02ff60 0x7f947c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f94780096a0 tx=0x7f9478006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:38.780 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.789+0000 7f948cff9700 1 -- 172.21.15.194:0/3763181886 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9480012920 con 0x7f9490060220 2024-01-27T23:29:38.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.995+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f949010a0e0 con 0x7f9490060220 2024-01-27T23:29:38.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.996+0000 7f948cff9700 1 -- 172.21.15.194:0/3763181886 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9480012330 con 0x7f9490060220 2024-01-27T23:29:38.988 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:38.988 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:38.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f947c02ff60 msgr2=0x7f947c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:38.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f947c02ff60 0x7f947c032420 secure :-1 s=READY pgs=93 cs=0 l=1 rev1=1 rx=0x7f94780096a0 tx=0x7f9478006b40).stop 2024-01-27T23:29:38.989 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 msgr2=0x7f94901088b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94901088b0 secure :-1 s=READY pgs=254 cs=0 l=1 rev1=1 rx=0x7f9480013510 tx=0x7f948000b040).stop 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 shutdown_connections 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f947c02ff60 0x7f947c032420 unknown :-1 s=CLOSED pgs=93 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 --2- 172.21.15.194:0/3763181886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9490060220 0x7f94901088b0 unknown :-1 s=CLOSED pgs=254 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.997+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 >> 172.21.15.194:0/3763181886 conn(0x7f94900f0b70 msgr2=0x7f94900f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.998+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 shutdown_connections 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:38.998+0000 7f94968d7700 1 -- 172.21.15.194:0/3763181886 wait complete. 2024-01-27T23:29:38.990 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:39.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:39 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:38.996908+0000 mon.smithi121 (mon.0) 208 : audit [DBG] from='client.? 172.21.15.194:0/3763181886' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:40.372 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:40.372 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:40.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:39.297844+0000 mgr.smithi121.hcppvz (mgr.14162) 132 : cluster [DBG] pgmap v125: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:41.492 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 -- 172.21.15.194:0/3653772445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 msgr2=0x7f63b40f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 --2- 172.21.15.194:0/3653772445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b40f6930 secure :-1 s=READY pgs=255 cs=0 l=1 rev1=1 rx=0x7f63a4009fb0 tx=0x7f63a4009b30).stop 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 -- 172.21.15.194:0/3653772445 shutdown_connections 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 --2- 172.21.15.194:0/3653772445 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b40f6930 unknown :-1 s=CLOSED pgs=255 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 -- 172.21.15.194:0/3653772445 >> 172.21.15.194:0/3653772445 conn(0x7f63b40f1c70 msgr2=0x7f63b40f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 -- 172.21.15.194:0/3653772445 shutdown_connections 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 -- 172.21.15.194:0/3653772445 wait complete. 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.500+0000 7f63baa23700 1 Processor -- start 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.501+0000 7f63baa23700 1 -- start start 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.501+0000 7f63baa23700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b41066d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:41.493 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.501+0000 7f63baa23700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f63b4106bc0 con 0x7f63b40f6510 2024-01-27T23:29:41.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.501+0000 7f63b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b41066d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:41.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.501+0000 7f63b3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b41066d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37532/0 (socket says 172.21.15.194:37532) 2024-01-27T23:29:41.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.501+0000 7f63b3fff700 1 -- 172.21.15.194:0/3144550551 learned_addr learned my addr 172.21.15.194:0/3144550551 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:41.495 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.501+0000 7f63b3fff700 1 -- 172.21.15.194:0/3144550551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f63a4005040 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.502+0000 7f63b3fff700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b41066d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f63a4013510 tx=0x7f63a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.502+0000 7f63b0ff9700 1 -- 172.21.15.194:0/3144550551 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a4009320 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.502+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f63b4106dc0 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.502+0000 7f63b0ff9700 1 -- 172.21.15.194:0/3144550551 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f63a4013f00 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.502+0000 7f63b0ff9700 1 -- 172.21.15.194:0/3144550551 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f63a4014d90 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.502+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f63b4107250 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.502+0000 7f63b0ff9700 1 -- 172.21.15.194:0/3144550551 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f63a40263a0 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.503+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f63b4045ad0 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.503+0000 7f63b0ff9700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f639c02ffb0 0x7f639c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.503+0000 7f63b0ff9700 1 -- 172.21.15.194:0/3144550551 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f63a4041340 con 0x7f63b40f6510 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.503+0000 7f63b37fe700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f639c02ffb0 0x7f639c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:41.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.504+0000 7f63b37fe700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f639c02ffb0 0x7f639c032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f63a80096a0 tx=0x7f63a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:41.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.508+0000 7f63b0ff9700 1 -- 172.21.15.194:0/3144550551 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f63a401c920 con 0x7f63b40f6510 2024-01-27T23:29:41.707 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.715+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f63b4108100 con 0x7f63b40f6510 2024-01-27T23:29:41.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.717+0000 7f63b0ff9700 1 -- 172.21.15.194:0/3144550551 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f63a401c920 con 0x7f63b40f6510 2024-01-27T23:29:41.708 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:41.708 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:41.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f639c02ffb0 msgr2=0x7f639c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f639c02ffb0 0x7f639c032470 secure :-1 s=READY pgs=94 cs=0 l=1 rev1=1 rx=0x7f63a80096a0 tx=0x7f63a8006b40).stop 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 msgr2=0x7f63b41066d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b41066d0 secure :-1 s=READY pgs=256 cs=0 l=1 rev1=1 rx=0x7f63a4013510 tx=0x7f63a400b040).stop 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 shutdown_connections 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f639c02ffb0 0x7f639c032470 unknown :-1 s=CLOSED pgs=94 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 --2- 172.21.15.194:0/3144550551 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f63b40f6510 0x7f63b41066d0 unknown :-1 s=CLOSED pgs=256 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 >> 172.21.15.194:0/3144550551 conn(0x7f63b40f1c70 msgr2=0x7f63b40f2710 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 shutdown_connections 2024-01-27T23:29:41.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:41.718+0000 7f63baa23700 1 -- 172.21.15.194:0/3144550551 wait complete. 2024-01-27T23:29:41.711 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:42.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:41.298249+0000 mgr.smithi121.hcppvz (mgr.14162) 133 : cluster [DBG] pgmap v126: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:42.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:41.717159+0000 mon.smithi121 (mon.0) 209 : audit [DBG] from='client.? 172.21.15.194:0/3144550551' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:43.131 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:43.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:44.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.238+0000 7ff07863f700 1 -- 172.21.15.194:0/1605781558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 msgr2=0x7ff0700f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.238+0000 7ff07863f700 1 --2- 172.21.15.194:0/1605781558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0700f90f0 secure :-1 s=READY pgs=257 cs=0 l=1 rev1=1 rx=0x7ff060009fb0 tx=0x7ff060009b30).stop 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.238+0000 7ff07863f700 1 -- 172.21.15.194:0/1605781558 shutdown_connections 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.238+0000 7ff07863f700 1 --2- 172.21.15.194:0/1605781558 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0700f90f0 unknown :-1 s=CLOSED pgs=257 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.238+0000 7ff07863f700 1 -- 172.21.15.194:0/1605781558 >> 172.21.15.194:0/1605781558 conn(0x7ff0700f0b70 msgr2=0x7ff0700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.238+0000 7ff07863f700 1 -- 172.21.15.194:0/1605781558 shutdown_connections 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.238+0000 7ff07863f700 1 -- 172.21.15.194:0/1605781558 wait complete. 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff07863f700 1 Processor -- start 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff07863f700 1 -- start start 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff07863f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0701066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:44.231 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff07863f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff070106bd0 con 0x7ff070060280 2024-01-27T23:29:44.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff0763db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0701066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:44.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff0763db700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0701066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37554/0 (socket says 172.21.15.194:37554) 2024-01-27T23:29:44.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff0763db700 1 -- 172.21.15.194:0/4142405610 learned_addr learned my addr 172.21.15.194:0/4142405610 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:44.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.239+0000 7ff0763db700 1 -- 172.21.15.194:0/4142405610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff060005040 con 0x7ff070060280 2024-01-27T23:29:44.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.240+0000 7ff0763db700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0701066e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7ff060013510 tx=0x7ff06000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:44.233 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.240+0000 7ff066ffd700 1 -- 172.21.15.194:0/4142405610 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff060009320 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.240+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff070106dd0 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.240+0000 7ff066ffd700 1 -- 172.21.15.194:0/4142405610 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff060013f00 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.240+0000 7ff066ffd700 1 -- 172.21.15.194:0/4142405610 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff060014dc0 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.240+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff070107260 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.241+0000 7ff066ffd700 1 -- 172.21.15.194:0/4142405610 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff060014420 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.241+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff070045ad0 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.241+0000 7ff066ffd700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff05c02ff60 0x7ff05c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.241+0000 7ff066ffd700 1 -- 172.21.15.194:0/4142405610 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff060041030 con 0x7ff070060280 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.241+0000 7ff075bda700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff05c02ff60 0x7ff05c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:44.234 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.241+0000 7ff075bda700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff05c02ff60 0x7ff05c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff06c0096a0 tx=0x7ff06c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:44.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.246+0000 7ff066ffd700 1 -- 172.21.15.194:0/4142405610 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff0600095f0 con 0x7ff070060280 2024-01-27T23:29:44.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.455+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff070108110 con 0x7ff070060280 2024-01-27T23:29:44.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.456+0000 7ff066ffd700 1 -- 172.21.15.194:0/4142405610 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff06001cb10 con 0x7ff070060280 2024-01-27T23:29:44.448 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:44.448 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff05c02ff60 msgr2=0x7ff05c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff05c02ff60 0x7ff05c032420 secure :-1 s=READY pgs=95 cs=0 l=1 rev1=1 rx=0x7ff06c0096a0 tx=0x7ff06c006b40).stop 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 msgr2=0x7ff0701066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0701066e0 secure :-1 s=READY pgs=258 cs=0 l=1 rev1=1 rx=0x7ff060013510 tx=0x7ff06000b040).stop 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 shutdown_connections 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff05c02ff60 0x7ff05c032420 unknown :-1 s=CLOSED pgs=95 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 --2- 172.21.15.194:0/4142405610 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff070060280 0x7ff0701066e0 unknown :-1 s=CLOSED pgs=258 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 >> 172.21.15.194:0/4142405610 conn(0x7ff0700f0b70 msgr2=0x7ff0700f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:44.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 shutdown_connections 2024-01-27T23:29:44.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:44.458+0000 7ff07863f700 1 -- 172.21.15.194:0/4142405610 wait complete. 2024-01-27T23:29:44.451 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:44.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:43.298467+0000 mgr.smithi121.hcppvz (mgr.14162) 134 : cluster [DBG] pgmap v127: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:45.835 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:45.835 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:45.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:45 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:44.457203+0000 mon.smithi121 (mon.0) 210 : audit [DBG] from='client.? 172.21.15.194:0/4142405610' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:46.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:45.298740+0000 mgr.smithi121.hcppvz (mgr.14162) 135 : cluster [DBG] pgmap v128: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:47.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 -- 172.21.15.194:0/4240193340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 msgr2=0x7f17e40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:47.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 --2- 172.21.15.194:0/4240193340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e40f66c0 secure :-1 s=READY pgs=259 cs=0 l=1 rev1=1 rx=0x7f17d4009fb0 tx=0x7f17d4009b30).stop 2024-01-27T23:29:47.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 -- 172.21.15.194:0/4240193340 shutdown_connections 2024-01-27T23:29:47.063 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 --2- 172.21.15.194:0/4240193340 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e40f66c0 unknown :-1 s=CLOSED pgs=259 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 -- 172.21.15.194:0/4240193340 >> 172.21.15.194:0/4240193340 conn(0x7f17e40f1aa0 msgr2=0x7f17e40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 -- 172.21.15.194:0/4240193340 shutdown_connections 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 -- 172.21.15.194:0/4240193340 wait complete. 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 Processor -- start 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 -- start start 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e406b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.071+0000 7f17e359e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f17e4069e20 con 0x7f17e40f62a0 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17e259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e406b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17e259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e406b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55754/0 (socket says 172.21.15.194:55754) 2024-01-27T23:29:47.064 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17e259c700 1 -- 172.21.15.194:0/3233797362 learned_addr learned my addr 172.21.15.194:0/3233797362 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17e259c700 1 -- 172.21.15.194:0/3233797362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f17d4005040 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17e259c700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e406b9a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f17d4013510 tx=0x7f17d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17d2ffd700 1 -- 172.21.15.194:0/3233797362 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17d4009320 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f17e4069fc0 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17d2ffd700 1 -- 172.21.15.194:0/3233797362 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f17d4013f00 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17d2ffd700 1 -- 172.21.15.194:0/3233797362 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f17d4014dc0 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.072+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f17e406a450 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.073+0000 7f17d2ffd700 1 -- 172.21.15.194:0/3233797362 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f17d40095f0 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.073+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f17e4059100 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.073+0000 7f17d2ffd700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17cc02ff60 0x7f17cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.073+0000 7f17d2ffd700 1 -- 172.21.15.194:0/3233797362 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f17d40410c0 con 0x7f17e40f62a0 2024-01-27T23:29:47.066 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.073+0000 7f17e1d9b700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17cc02ff60 0x7f17cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:47.067 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.074+0000 7f17e1d9b700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17cc02ff60 0x7f17cc032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f17d80096a0 tx=0x7f17d8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:47.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.079+0000 7f17d2ffd700 1 -- 172.21.15.194:0/3233797362 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f17d401c790 con 0x7f17e40f62a0 2024-01-27T23:29:47.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.283+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f17e40fbce0 con 0x7f17e40f62a0 2024-01-27T23:29:47.275 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.283+0000 7f17d2ffd700 1 -- 172.21.15.194:0/3233797362 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f17d4010230 con 0x7f17e40f62a0 2024-01-27T23:29:47.275 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:47.275 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17cc02ff60 msgr2=0x7f17cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17cc02ff60 0x7f17cc032420 secure :-1 s=READY pgs=96 cs=0 l=1 rev1=1 rx=0x7f17d80096a0 tx=0x7f17d8006b40).stop 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 msgr2=0x7f17e406b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e406b9a0 secure :-1 s=READY pgs=260 cs=0 l=1 rev1=1 rx=0x7f17d4013510 tx=0x7f17d400b040).stop 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 shutdown_connections 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f17cc02ff60 0x7f17cc032420 unknown :-1 s=CLOSED pgs=96 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 --2- 172.21.15.194:0/3233797362 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f17e40f62a0 0x7f17e406b9a0 unknown :-1 s=CLOSED pgs=260 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 >> 172.21.15.194:0/3233797362 conn(0x7f17e40f1aa0 msgr2=0x7f17e40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 shutdown_connections 2024-01-27T23:29:47.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:47.285+0000 7f17e359e700 1 -- 172.21.15.194:0/3233797362 wait complete. 2024-01-27T23:29:47.278 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:47.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:47 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:47.284535+0000 mon.smithi121 (mon.0) 211 : audit [DBG] from='client.? 172.21.15.194:0/3233797362' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:48.650 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:48.651 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:48.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:47.299039+0000 mgr.smithi121.hcppvz (mgr.14162) 136 : cluster [DBG] pgmap v129: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:49.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.721+0000 7f98dcfd1700 1 -- 172.21.15.194:0/1716217735 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 msgr2=0x7f98d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:49.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.721+0000 7f98dcfd1700 1 --2- 172.21.15.194:0/1716217735 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80f5830 secure :-1 s=READY pgs=261 cs=0 l=1 rev1=1 rx=0x7f98c8009fb0 tx=0x7f98c8009b30).stop 2024-01-27T23:29:49.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.721+0000 7f98dcfd1700 1 -- 172.21.15.194:0/1716217735 shutdown_connections 2024-01-27T23:29:49.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.721+0000 7f98dcfd1700 1 --2- 172.21.15.194:0/1716217735 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80f5830 unknown :-1 s=CLOSED pgs=261 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:49.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.721+0000 7f98dcfd1700 1 -- 172.21.15.194:0/1716217735 >> 172.21.15.194:0/1716217735 conn(0x7f98d80f0b70 msgr2=0x7f98d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.721+0000 7f98dcfd1700 1 -- 172.21.15.194:0/1716217735 shutdown_connections 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98dcfd1700 1 -- 172.21.15.194:0/1716217735 wait complete. 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98dcfd1700 1 Processor -- start 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98dcfd1700 1 -- start start 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98dcfd1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98dcfd1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f98d80fb820 con 0x7f98d80f5410 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98d659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98d659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55768/0 (socket says 172.21.15.194:55768) 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.722+0000 7f98d659c700 1 -- 172.21.15.194:0/215380832 learned_addr learned my addr 172.21.15.194:0/215380832 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.723+0000 7f98d659c700 1 -- 172.21.15.194:0/215380832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f98c8005040 con 0x7f98d80f5410 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.723+0000 7f98d659c700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80fd3a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f98c8013510 tx=0x7f98c800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.723+0000 7f98c6ffd700 1 -- 172.21.15.194:0/215380832 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98c8009320 con 0x7f98d80f5410 2024-01-27T23:29:49.716 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.723+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f98d80fb9c0 con 0x7f98d80f5410 2024-01-27T23:29:49.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.723+0000 7f98c6ffd700 1 -- 172.21.15.194:0/215380832 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f98c8013f00 con 0x7f98d80f5410 2024-01-27T23:29:49.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.723+0000 7f98c6ffd700 1 -- 172.21.15.194:0/215380832 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f98c8014d90 con 0x7f98d80f5410 2024-01-27T23:29:49.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.723+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f98d80fbe50 con 0x7f98d80f5410 2024-01-27T23:29:49.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.724+0000 7f98c6ffd700 1 -- 172.21.15.194:0/215380832 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f98c80095f0 con 0x7f98d80f5410 2024-01-27T23:29:49.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.724+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f98d8059100 con 0x7f98d80f5410 2024-01-27T23:29:49.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.724+0000 7f98c6ffd700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f98c002ff60 0x7f98c0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:49.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.724+0000 7f98c6ffd700 1 -- 172.21.15.194:0/215380832 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f98c8041000 con 0x7f98d80f5410 2024-01-27T23:29:49.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.724+0000 7f98d5d9b700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f98c002ff60 0x7f98c0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:49.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.724+0000 7f98d5d9b700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f98c002ff60 0x7f98c0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f98cc0096a0 tx=0x7f98cc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:49.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.729+0000 7f98c6ffd700 1 -- 172.21.15.194:0/215380832 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f98c8014420 con 0x7f98d80f5410 2024-01-27T23:29:49.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.935+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f98d8045ad0 con 0x7f98d80f5410 2024-01-27T23:29:49.928 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.937+0000 7f98c6ffd700 1 -- 172.21.15.194:0/215380832 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f98c8010230 con 0x7f98d80f5410 2024-01-27T23:29:49.928 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:49.928 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f98c002ff60 msgr2=0x7f98c0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f98c002ff60 0x7f98c0032420 secure :-1 s=READY pgs=97 cs=0 l=1 rev1=1 rx=0x7f98cc0096a0 tx=0x7f98cc006b40).stop 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 msgr2=0x7f98d80fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80fd3a0 secure :-1 s=READY pgs=262 cs=0 l=1 rev1=1 rx=0x7f98c8013510 tx=0x7f98c800b040).stop 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 shutdown_connections 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f98c002ff60 0x7f98c0032420 unknown :-1 s=CLOSED pgs=97 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 --2- 172.21.15.194:0/215380832 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f98d80f5410 0x7f98d80fd3a0 unknown :-1 s=CLOSED pgs=262 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 >> 172.21.15.194:0/215380832 conn(0x7f98d80f0b70 msgr2=0x7f98d805fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 shutdown_connections 2024-01-27T23:29:49.930 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:49.938+0000 7f98dcfd1700 1 -- 172.21.15.194:0/215380832 wait complete. 2024-01-27T23:29:49.931 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:50.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:49.299339+0000 mgr.smithi121.hcppvz (mgr.14162) 137 : cluster [DBG] pgmap v130: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:50.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:49.937194+0000 mon.smithi121 (mon.0) 212 : audit [DBG] from='client.? 172.21.15.194:0/215380832' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:51.308 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:51.308 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:52.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.458+0000 7f29b5384700 1 -- 172.21.15.194:0/3494424376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 msgr2=0x7f29b00f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:52.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.458+0000 7f29b5384700 1 --2- 172.21.15.194:0/3494424376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f9150 secure :-1 s=READY pgs=263 cs=0 l=1 rev1=1 rx=0x7f29a4009fb0 tx=0x7f29a4009b30).stop 2024-01-27T23:29:52.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.458+0000 7f29b5384700 1 -- 172.21.15.194:0/3494424376 shutdown_connections 2024-01-27T23:29:52.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.458+0000 7f29b5384700 1 --2- 172.21.15.194:0/3494424376 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f9150 unknown :-1 s=CLOSED pgs=263 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:52.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.458+0000 7f29b5384700 1 -- 172.21.15.194:0/3494424376 >> 172.21.15.194:0/3494424376 conn(0x7f29b00f0b70 msgr2=0x7f29b00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:52.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.458+0000 7f29b5384700 1 -- 172.21.15.194:0/3494424376 shutdown_connections 2024-01-27T23:29:52.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.458+0000 7f29b5384700 1 -- 172.21.15.194:0/3494424376 wait complete. 2024-01-27T23:29:52.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.459+0000 7f29b5384700 1 Processor -- start 2024-01-27T23:29:52.452 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.459+0000 7f29b5384700 1 -- start start 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.459+0000 7f29b5384700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f6bd0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.459+0000 7f29b5384700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29b00f5050 con 0x7f29b0060220 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29aeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f6bd0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29aeffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f6bd0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55778/0 (socket says 172.21.15.194:55778) 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29aeffd700 1 -- 172.21.15.194:0/2629423394 learned_addr learned my addr 172.21.15.194:0/2629423394 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29aeffd700 1 -- 172.21.15.194:0/2629423394 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4005040 con 0x7f29b0060220 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29aeffd700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f6bd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f29a4009830 tx=0x7f29a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29977fe700 1 -- 172.21.15.194:0/2629423394 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29a4009320 con 0x7f29b0060220 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29977fe700 1 -- 172.21.15.194:0/2629423394 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29a40136c0 con 0x7f29b0060220 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29b00f51f0 con 0x7f29b0060220 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29977fe700 1 -- 172.21.15.194:0/2629423394 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29a401ee90 con 0x7f29b0060220 2024-01-27T23:29:52.453 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.460+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29b00f5680 con 0x7f29b0060220 2024-01-27T23:29:52.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.461+0000 7f29977fe700 1 -- 172.21.15.194:0/2629423394 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29a401e420 con 0x7f29b0060220 2024-01-27T23:29:52.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.461+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29b0059100 con 0x7f29b0060220 2024-01-27T23:29:52.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.461+0000 7f29977fe700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f299802ff10 0x7f29980323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:52.454 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.461+0000 7f29977fe700 1 -- 172.21.15.194:0/2629423394 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29a4041040 con 0x7f29b0060220 2024-01-27T23:29:52.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.461+0000 7f29ae7fc700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f299802ff10 0x7f29980323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:52.455 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.462+0000 7f29ae7fc700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f299802ff10 0x7f29980323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f29a00096a0 tx=0x7f29a0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:52.458 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.466+0000 7f29977fe700 1 -- 172.21.15.194:0/2629423394 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29a406d2b0 con 0x7f29b0060220 2024-01-27T23:29:52.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:51.299713+0000 mgr.smithi121.hcppvz (mgr.14162) 138 : cluster [DBG] pgmap v131: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:52.663 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.671+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29b00fbf50 con 0x7f29b0060220 2024-01-27T23:29:52.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.672+0000 7f29977fe700 1 -- 172.21.15.194:0/2629423394 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f29a401c790 con 0x7f29b0060220 2024-01-27T23:29:52.664 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:52.664 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f299802ff10 msgr2=0x7f29980323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f299802ff10 0x7f29980323d0 secure :-1 s=READY pgs=98 cs=0 l=1 rev1=1 rx=0x7f29a00096a0 tx=0x7f29a0006b40).stop 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 msgr2=0x7f29b00f6bd0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f6bd0 secure :-1 s=READY pgs=264 cs=0 l=1 rev1=1 rx=0x7f29a4009830 tx=0x7f29a400b040).stop 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 shutdown_connections 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f299802ff10 0x7f29980323d0 unknown :-1 s=CLOSED pgs=98 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 --2- 172.21.15.194:0/2629423394 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29b0060220 0x7f29b00f6bd0 unknown :-1 s=CLOSED pgs=264 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 >> 172.21.15.194:0/2629423394 conn(0x7f29b00f0b70 msgr2=0x7f29b00f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 shutdown_connections 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:52.674+0000 7f29b5384700 1 -- 172.21.15.194:0/2629423394 wait complete. 2024-01-27T23:29:52.666 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:53.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:53 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:52.673462+0000 mon.smithi121 (mon.0) 213 : audit [DBG] from='client.? 172.21.15.194:0/2629423394' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:54.062 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:54.062 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:54.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:53.299982+0000 mgr.smithi121.hcppvz (mgr.14162) 139 : cluster [DBG] pgmap v132: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:55.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.194+0000 7f2f73417700 1 -- 172.21.15.194:0/881315730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 msgr2=0x7f2f6c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:55.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.194+0000 7f2f73417700 1 --2- 172.21.15.194:0/881315730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c0f5830 secure :-1 s=READY pgs=265 cs=0 l=1 rev1=1 rx=0x7f2f5c009fb0 tx=0x7f2f5c009b30).stop 2024-01-27T23:29:55.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.194+0000 7f2f73417700 1 -- 172.21.15.194:0/881315730 shutdown_connections 2024-01-27T23:29:55.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.194+0000 7f2f73417700 1 --2- 172.21.15.194:0/881315730 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c0f5830 unknown :-1 s=CLOSED pgs=265 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.194+0000 7f2f73417700 1 -- 172.21.15.194:0/881315730 >> 172.21.15.194:0/881315730 conn(0x7f2f6c0f0b70 msgr2=0x7f2f6c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.194+0000 7f2f73417700 1 -- 172.21.15.194:0/881315730 shutdown_connections 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.194+0000 7f2f73417700 1 -- 172.21.15.194:0/881315730 wait complete. 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f73417700 1 Processor -- start 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f73417700 1 -- start start 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f73417700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c106760 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f73417700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2f6c106c50 con 0x7f2f6c0f5410 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f711b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c106760 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f711b3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c106760 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55784/0 (socket says 172.21.15.194:55784) 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f711b3700 1 -- 172.21.15.194:0/1215792909 learned_addr learned my addr 172.21.15.194:0/1215792909 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.195+0000 7f2f711b3700 1 -- 172.21.15.194:0/1215792909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2f5c005040 con 0x7f2f6c0f5410 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.196+0000 7f2f711b3700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c106760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f2f5c0089f0 tx=0x7f2f5c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.196+0000 7f2f61ffb700 1 -- 172.21.15.194:0/1215792909 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f5c009320 con 0x7f2f6c0f5410 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.196+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2f6c106e50 con 0x7f2f6c0f5410 2024-01-27T23:29:55.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.196+0000 7f2f61ffb700 1 -- 172.21.15.194:0/1215792909 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2f5c0136c0 con 0x7f2f6c0f5410 2024-01-27T23:29:55.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.196+0000 7f2f61ffb700 1 -- 172.21.15.194:0/1215792909 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2f5c01ecb0 con 0x7f2f6c0f5410 2024-01-27T23:29:55.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.196+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2f6c1072e0 con 0x7f2f6c0f5410 2024-01-27T23:29:55.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.197+0000 7f2f61ffb700 1 -- 172.21.15.194:0/1215792909 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2f5c014420 con 0x7f2f6c0f5410 2024-01-27T23:29:55.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.197+0000 7f2f61ffb700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2f5802ff60 0x7f2f58032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:55.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.197+0000 7f2f61ffb700 1 -- 172.21.15.194:0/1215792909 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2f5c041190 con 0x7f2f6c0f5410 2024-01-27T23:29:55.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.197+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2f6c045ad0 con 0x7f2f6c0f5410 2024-01-27T23:29:55.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.197+0000 7f2f709b2700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2f5802ff60 0x7f2f58032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:55.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.198+0000 7f2f709b2700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2f5802ff60 0x7f2f58032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2f680096a0 tx=0x7f2f68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:55.193 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.202+0000 7f2f61ffb700 1 -- 172.21.15.194:0/1215792909 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2f5c01c910 con 0x7f2f6c0f5410 2024-01-27T23:29:55.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.406+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2f6c108000 con 0x7f2f6c0f5410 2024-01-27T23:29:55.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.407+0000 7f2f61ffb700 1 -- 172.21.15.194:0/1215792909 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2f5c01c320 con 0x7f2f6c0f5410 2024-01-27T23:29:55.399 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:55.399 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:55.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2f5802ff60 msgr2=0x7f2f58032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:55.400 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2f5802ff60 0x7f2f58032420 secure :-1 s=READY pgs=99 cs=0 l=1 rev1=1 rx=0x7f2f680096a0 tx=0x7f2f68006b40).stop 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 msgr2=0x7f2f6c106760 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c106760 secure :-1 s=READY pgs=266 cs=0 l=1 rev1=1 rx=0x7f2f5c0089f0 tx=0x7f2f5c00b040).stop 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 shutdown_connections 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2f5802ff60 0x7f2f58032420 unknown :-1 s=CLOSED pgs=99 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 --2- 172.21.15.194:0/1215792909 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2f6c0f5410 0x7f2f6c106760 unknown :-1 s=CLOSED pgs=266 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 >> 172.21.15.194:0/1215792909 conn(0x7f2f6c0f0b70 msgr2=0x7f2f6c05fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 shutdown_connections 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:55.408+0000 7f2f73417700 1 -- 172.21.15.194:0/1215792909 wait complete. 2024-01-27T23:29:55.401 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:56.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:56.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:29:56.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:55.300343+0000 mgr.smithi121.hcppvz (mgr.14162) 140 : cluster [DBG] pgmap v133: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:56.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:55.408179+0000 mon.smithi121 (mon.0) 214 : audit [DBG] from='client.? 172.21.15.194:0/1215792909' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:57.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 -- 172.21.15.194:0/2258792922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 msgr2=0x7fd4180f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:57.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 --2- 172.21.15.194:0/2258792922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4180f66c0 secure :-1 s=READY pgs=267 cs=0 l=1 rev1=1 rx=0x7fd408009fb0 tx=0x7fd408009b30).stop 2024-01-27T23:29:57.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 -- 172.21.15.194:0/2258792922 shutdown_connections 2024-01-27T23:29:57.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 --2- 172.21.15.194:0/2258792922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4180f66c0 unknown :-1 s=CLOSED pgs=267 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:57.903 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 -- 172.21.15.194:0/2258792922 >> 172.21.15.194:0/2258792922 conn(0x7fd4180f1aa0 msgr2=0x7fd4180f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 -- 172.21.15.194:0/2258792922 shutdown_connections 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 -- 172.21.15.194:0/2258792922 wait complete. 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 Processor -- start 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 -- start start 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4181064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.910+0000 7fd41f106700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd4181069a0 con 0x7fd4180f62a0 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd41e104700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4181064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd41e104700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4181064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:49052/0 (socket says 172.21.15.194:49052) 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd41e104700 1 -- 172.21.15.194:0/3181196618 learned_addr learned my addr 172.21.15.194:0/3181196618 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd41e104700 1 -- 172.21.15.194:0/3181196618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd408005040 con 0x7fd4180f62a0 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd41e104700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4181064b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fd408013510 tx=0x7fd40800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd40effd700 1 -- 172.21.15.194:0/3181196618 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd408009320 con 0x7fd4180f62a0 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd418106ba0 con 0x7fd4180f62a0 2024-01-27T23:29:57.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd40effd700 1 -- 172.21.15.194:0/3181196618 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd408013f00 con 0x7fd4180f62a0 2024-01-27T23:29:57.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd40effd700 1 -- 172.21.15.194:0/3181196618 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd408014dc0 con 0x7fd4180f62a0 2024-01-27T23:29:57.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.911+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd418107030 con 0x7fd4180f62a0 2024-01-27T23:29:57.905 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.912+0000 7fd40effd700 1 -- 172.21.15.194:0/3181196618 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd4080095f0 con 0x7fd4180f62a0 2024-01-27T23:29:57.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.912+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd418045ad0 con 0x7fd4180f62a0 2024-01-27T23:29:57.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.912+0000 7fd40effd700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd40402ff60 0x7fd404032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:29:57.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.912+0000 7fd40effd700 1 -- 172.21.15.194:0/3181196618 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd408041000 con 0x7fd4180f62a0 2024-01-27T23:29:57.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.912+0000 7fd41d903700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd40402ff60 0x7fd404032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:29:57.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.913+0000 7fd41d903700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd40402ff60 0x7fd404032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fd4140096a0 tx=0x7fd414006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:29:57.909 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:57.917+0000 7fd40effd700 1 -- 172.21.15.194:0/3181196618 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd408014420 con 0x7fd4180f62a0 2024-01-27T23:29:58.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.126+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd418107ee0 con 0x7fd4180f62a0 2024-01-27T23:29:58.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd40effd700 1 -- 172.21.15.194:0/3181196618 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd408010230 con 0x7fd4180f62a0 2024-01-27T23:29:58.119 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:29:58.119 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:29:58.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd40402ff60 msgr2=0x7fd404032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:58.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd40402ff60 0x7fd404032420 secure :-1 s=READY pgs=100 cs=0 l=1 rev1=1 rx=0x7fd4140096a0 tx=0x7fd414006b40).stop 2024-01-27T23:29:58.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 msgr2=0x7fd4181064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:29:58.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4181064b0 secure :-1 s=READY pgs=268 cs=0 l=1 rev1=1 rx=0x7fd408013510 tx=0x7fd40800b040).stop 2024-01-27T23:29:58.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 shutdown_connections 2024-01-27T23:29:58.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd40402ff60 0x7fd404032420 unknown :-1 s=CLOSED pgs=100 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:58.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 --2- 172.21.15.194:0/3181196618 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd4180f62a0 0x7fd4181064b0 unknown :-1 s=CLOSED pgs=268 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:29:58.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 >> 172.21.15.194:0/3181196618 conn(0x7fd4180f1aa0 msgr2=0x7fd4180f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:29:58.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 shutdown_connections 2024-01-27T23:29:58.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:29:58.128+0000 7fd41f106700 1 -- 172.21.15.194:0/3181196618 wait complete. 2024-01-27T23:29:58.121 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:29:58.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:57.300693+0000 mgr.smithi121.hcppvz (mgr.14162) 141 : cluster [DBG] pgmap v134: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:29:58.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:29:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:29:58.127914+0000 mon.smithi121 (mon.0) 215 : audit [DBG] from='client.? 172.21.15.194:0/3181196618' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:29:59.515 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:29:59.515 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:00.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 -- 172.21.15.194:0/3380815360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 msgr2=0x7f80f80f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 --2- 172.21.15.194:0/3380815360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f80f90f0 secure :-1 s=READY pgs=269 cs=0 l=1 rev1=1 rx=0x7f80ec009fb0 tx=0x7f80ec009b30).stop 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 -- 172.21.15.194:0/3380815360 shutdown_connections 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 --2- 172.21.15.194:0/3380815360 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f80f90f0 unknown :-1 s=CLOSED pgs=269 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 -- 172.21.15.194:0/3380815360 >> 172.21.15.194:0/3380815360 conn(0x7f80f80f0b70 msgr2=0x7f80f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 -- 172.21.15.194:0/3380815360 shutdown_connections 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 -- 172.21.15.194:0/3380815360 wait complete. 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 Processor -- start 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 -- start start 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80feced700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f80f8106bd0 con 0x7f80f8060280 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80fca89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80fca89700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:49074/0 (socket says 172.21.15.194:49074) 2024-01-27T23:30:00.613 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.620+0000 7f80fca89700 1 -- 172.21.15.194:0/660251115 learned_addr learned my addr 172.21.15.194:0/660251115 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:00.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.621+0000 7f80fca89700 1 -- 172.21.15.194:0/660251115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f80ec005040 con 0x7f80f8060280 2024-01-27T23:30:00.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.621+0000 7f80fca89700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f81066e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f80ec013510 tx=0x7f80ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:00.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.621+0000 7f80f57fa700 1 -- 172.21.15.194:0/660251115 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ec009320 con 0x7f80f8060280 2024-01-27T23:30:00.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.621+0000 7f80f57fa700 1 -- 172.21.15.194:0/660251115 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f80ec013f00 con 0x7f80f8060280 2024-01-27T23:30:00.615 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.621+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f80f8106dd0 con 0x7f80f8060280 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.621+0000 7f80f57fa700 1 -- 172.21.15.194:0/660251115 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f80ec014dc0 con 0x7f80f8060280 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.621+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f80f8107260 con 0x7f80f8060280 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.622+0000 7f80f57fa700 1 -- 172.21.15.194:0/660251115 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f80ec014420 con 0x7f80f8060280 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.622+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f80f8045ad0 con 0x7f80f8060280 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.622+0000 7f80f57fa700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80e802ffb0 0x7f80e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.622+0000 7f80f57fa700 1 -- 172.21.15.194:0/660251115 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f80ec041040 con 0x7f80f8060280 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.622+0000 7f80f7fff700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80e802ffb0 0x7f80e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:00.616 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.623+0000 7f80f7fff700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80e802ffb0 0x7f80e8032470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f80e40096a0 tx=0x7f80e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:00.618 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.627+0000 7f80f57fa700 1 -- 172.21.15.194:0/660251115 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f80ec0095f0 con 0x7f80f8060280 2024-01-27T23:30:00.828 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.836+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f80f8108110 con 0x7f80f8060280 2024-01-27T23:30:00.829 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.838+0000 7f80f57fa700 1 -- 172.21.15.194:0/660251115 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f80ec01db70 con 0x7f80f8060280 2024-01-27T23:30:00.829 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:00.829 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80e802ffb0 msgr2=0x7f80e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80e802ffb0 0x7f80e8032470 secure :-1 s=READY pgs=101 cs=0 l=1 rev1=1 rx=0x7f80e40096a0 tx=0x7f80e4006b40).stop 2024-01-27T23:30:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 msgr2=0x7f80f81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f81066e0 secure :-1 s=READY pgs=270 cs=0 l=1 rev1=1 rx=0x7f80ec013510 tx=0x7f80ec00b040).stop 2024-01-27T23:30:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 shutdown_connections 2024-01-27T23:30:00.831 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f80e802ffb0 0x7f80e8032470 unknown :-1 s=CLOSED pgs=101 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 --2- 172.21.15.194:0/660251115 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80f8060280 0x7f80f81066e0 unknown :-1 s=CLOSED pgs=270 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 >> 172.21.15.194:0/660251115 conn(0x7f80f80f0b70 msgr2=0x7f80f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 shutdown_connections 2024-01-27T23:30:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:00.839+0000 7f80feced700 1 -- 172.21.15.194:0/660251115 wait complete. 2024-01-27T23:30:00.832 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:00.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:29:59.300996+0000 mgr.smithi121.hcppvz (mgr.14162) 142 : cluster [DBG] pgmap v135: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:01.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:01 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:00.838263+0000 mon.smithi121 (mon.0) 216 : audit [DBG] from='client.? 172.21.15.194:0/660251115' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:02.256 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:02.256 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:02.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:01.301362+0000 mgr.smithi121.hcppvz (mgr.14162) 143 : cluster [DBG] pgmap v136: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:03.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.352+0000 7f530443d700 1 -- 172.21.15.194:0/1854875205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 msgr2=0x7f52fc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:03.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.352+0000 7f530443d700 1 --2- 172.21.15.194:0/1854875205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc0f6930 secure :-1 s=READY pgs=271 cs=0 l=1 rev1=1 rx=0x7f52ec009fb0 tx=0x7f52ec009b30).stop 2024-01-27T23:30:03.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.352+0000 7f530443d700 1 -- 172.21.15.194:0/1854875205 shutdown_connections 2024-01-27T23:30:03.347 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.352+0000 7f530443d700 1 --2- 172.21.15.194:0/1854875205 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc0f6930 unknown :-1 s=CLOSED pgs=271 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.352+0000 7f530443d700 1 -- 172.21.15.194:0/1854875205 >> 172.21.15.194:0/1854875205 conn(0x7f52fc0f1c70 msgr2=0x7f52fc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.352+0000 7f530443d700 1 -- 172.21.15.194:0/1854875205 shutdown_connections 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.352+0000 7f530443d700 1 -- 172.21.15.194:0/1854875205 wait complete. 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f530443d700 1 Processor -- start 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f530443d700 1 -- start start 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f530443d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f530443d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f52fc106c20 con 0x7f52fc0f6510 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f53021d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f53021d9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:49106/0 (socket says 172.21.15.194:49106) 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f53021d9700 1 -- 172.21.15.194:0/2849987399 learned_addr learned my addr 172.21.15.194:0/2849987399 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f53021d9700 1 -- 172.21.15.194:0/2849987399 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f52ec005040 con 0x7f52fc0f6510 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.353+0000 7f53021d9700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f52ec013510 tx=0x7f52ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.354+0000 7f52f2ffd700 1 -- 172.21.15.194:0/2849987399 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ec009320 con 0x7f52fc0f6510 2024-01-27T23:30:03.348 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.354+0000 7f52f2ffd700 1 -- 172.21.15.194:0/2849987399 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f52ec013f00 con 0x7f52fc0f6510 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.354+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f52fc106e20 con 0x7f52fc0f6510 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.354+0000 7f52f2ffd700 1 -- 172.21.15.194:0/2849987399 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f52ec014d90 con 0x7f52fc0f6510 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.354+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f52fc1072b0 con 0x7f52fc0f6510 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.354+0000 7f52f2ffd700 1 -- 172.21.15.194:0/2849987399 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f52ec014420 con 0x7f52fc0f6510 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.355+0000 7f52f2ffd700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f52e802ff60 0x7f52e8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.355+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f52fc045ad0 con 0x7f52fc0f6510 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.355+0000 7f52f2ffd700 1 -- 172.21.15.194:0/2849987399 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f52ec0410c0 con 0x7f52fc0f6510 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.355+0000 7f53019d8700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f52e802ff60 0x7f52e8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:03.349 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.355+0000 7f53019d8700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f52e802ff60 0x7f52e8032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f52f80096a0 tx=0x7f52f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:03.350 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.359+0000 7f52f2ffd700 1 -- 172.21.15.194:0/2849987399 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f52ec0095f0 con 0x7f52fc0f6510 2024-01-27T23:30:03.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.568+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f52fc0fc050 con 0x7f52fc0f6510 2024-01-27T23:30:03.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.569+0000 7f52f2ffd700 1 -- 172.21.15.194:0/2849987399 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f52ec02c090 con 0x7f52fc0f6510 2024-01-27T23:30:03.561 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:03.561 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f52e802ff60 msgr2=0x7f52e8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f52e802ff60 0x7f52e8032420 secure :-1 s=READY pgs=102 cs=0 l=1 rev1=1 rx=0x7f52f80096a0 tx=0x7f52f8006b40).stop 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 msgr2=0x7f52fc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc106730 secure :-1 s=READY pgs=272 cs=0 l=1 rev1=1 rx=0x7f52ec013510 tx=0x7f52ec00b040).stop 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 shutdown_connections 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f52e802ff60 0x7f52e8032420 unknown :-1 s=CLOSED pgs=102 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 --2- 172.21.15.194:0/2849987399 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f52fc0f6510 0x7f52fc106730 unknown :-1 s=CLOSED pgs=272 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 >> 172.21.15.194:0/2849987399 conn(0x7f52fc0f1c70 msgr2=0x7f52fc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 shutdown_connections 2024-01-27T23:30:03.563 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:03.571+0000 7f530443d700 1 -- 172.21.15.194:0/2849987399 wait complete. 2024-01-27T23:30:03.564 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:04.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:03.301667+0000 mgr.smithi121.hcppvz (mgr.14162) 144 : cluster [DBG] pgmap v137: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:04.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:03.569358+0000 mon.smithi121 (mon.0) 217 : audit [DBG] from='client.? 172.21.15.194:0/2849987399' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:04.936 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:04.936 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:06.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 -- 172.21.15.194:0/1958148452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 msgr2=0x7fb1240f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:06.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 --2- 172.21.15.194:0/1958148452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb1240f6db0 secure :-1 s=READY pgs=273 cs=0 l=1 rev1=1 rx=0x7fb120009fb0 tx=0x7fb120009b30).stop 2024-01-27T23:30:06.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 -- 172.21.15.194:0/1958148452 shutdown_connections 2024-01-27T23:30:06.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 --2- 172.21.15.194:0/1958148452 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb1240f6db0 unknown :-1 s=CLOSED pgs=273 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:06.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 -- 172.21.15.194:0/1958148452 >> 172.21.15.194:0/1958148452 conn(0x7fb1240f0260 msgr2=0x7fb1240f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:06.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 -- 172.21.15.194:0/1958148452 shutdown_connections 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 -- 172.21.15.194:0/1958148452 wait complete. 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 Processor -- start 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.094+0000 7fb12bb6b700 1 -- start start 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.095+0000 7fb12bb6b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb12410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.095+0000 7fb12bb6b700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb124107240 con 0x7fb1240f6990 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.095+0000 7fb129907700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb12410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.095+0000 7fb129907700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb12410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48154/0 (socket says 172.21.15.194:48154) 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.095+0000 7fb129907700 1 -- 172.21.15.194:0/2332205461 learned_addr learned my addr 172.21.15.194:0/2332205461 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.095+0000 7fb129907700 1 -- 172.21.15.194:0/2332205461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb120005040 con 0x7fb1240f6990 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.095+0000 7fb129907700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb12410b850 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb120013510 tx=0x7fb12000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.096+0000 7fb11a7fc700 1 -- 172.21.15.194:0/2332205461 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb120009320 con 0x7fb1240f6990 2024-01-27T23:30:06.088 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.096+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb1240f52b0 con 0x7fb1240f6990 2024-01-27T23:30:06.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.096+0000 7fb11a7fc700 1 -- 172.21.15.194:0/2332205461 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb120013f00 con 0x7fb1240f6990 2024-01-27T23:30:06.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.096+0000 7fb11a7fc700 1 -- 172.21.15.194:0/2332205461 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb120014d90 con 0x7fb1240f6990 2024-01-27T23:30:06.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.096+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb12410bd40 con 0x7fb1240f6990 2024-01-27T23:30:06.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.096+0000 7fb11a7fc700 1 -- 172.21.15.194:0/2332205461 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb1200263a0 con 0x7fb1240f6990 2024-01-27T23:30:06.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.096+0000 7fb11a7fc700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb11002ff60 0x7fb110032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:06.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.097+0000 7fb11a7fc700 1 -- 172.21.15.194:0/2332205461 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb1200410c0 con 0x7fb1240f6990 2024-01-27T23:30:06.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.097+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb124045ad0 con 0x7fb1240f6990 2024-01-27T23:30:06.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.097+0000 7fb129106700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb11002ff60 0x7fb110032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:06.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.097+0000 7fb129106700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb11002ff60 0x7fb110032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb1140096a0 tx=0x7fb114006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:06.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.101+0000 7fb11a7fc700 1 -- 172.21.15.194:0/2332205461 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb120012920 con 0x7fb1240f6990 2024-01-27T23:30:06.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.309+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb1240fae60 con 0x7fb1240f6990 2024-01-27T23:30:06.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.310+0000 7fb11a7fc700 1 -- 172.21.15.194:0/2332205461 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb1240fae60 con 0x7fb1240f6990 2024-01-27T23:30:06.303 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:06.303 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb11002ff60 msgr2=0x7fb110032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb11002ff60 0x7fb110032420 secure :-1 s=READY pgs=103 cs=0 l=1 rev1=1 rx=0x7fb1140096a0 tx=0x7fb114006b40).stop 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 msgr2=0x7fb12410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb12410b850 secure :-1 s=READY pgs=274 cs=0 l=1 rev1=1 rx=0x7fb120013510 tx=0x7fb12000b040).stop 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 shutdown_connections 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb11002ff60 0x7fb110032420 unknown :-1 s=CLOSED pgs=103 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 --2- 172.21.15.194:0/2332205461 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb1240f6990 0x7fb12410b850 unknown :-1 s=CLOSED pgs=274 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 >> 172.21.15.194:0/2332205461 conn(0x7fb1240f0260 msgr2=0x7fb1240f0d50 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 shutdown_connections 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:06.312+0000 7fb12bb6b700 1 -- 172.21.15.194:0/2332205461 wait complete. 2024-01-27T23:30:06.304 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:06.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:05.301970+0000 mgr.smithi121.hcppvz (mgr.14162) 145 : cluster [DBG] pgmap v138: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:06.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:06.310544+0000 mon.smithi121 (mon.0) 218 : audit [DBG] from='client.? 172.21.15.194:0/2332205461' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:07.691 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:07.691 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:08.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.842+0000 7f54a8557700 1 -- 172.21.15.194:0/4272037319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 msgr2=0x7f54a00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:08.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.842+0000 7f54a8557700 1 --2- 172.21.15.194:0/4272037319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00f5830 secure :-1 s=READY pgs=275 cs=0 l=1 rev1=1 rx=0x7f5490009fb0 tx=0x7f5490009b30).stop 2024-01-27T23:30:08.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.842+0000 7f54a8557700 1 -- 172.21.15.194:0/4272037319 shutdown_connections 2024-01-27T23:30:08.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.842+0000 7f54a8557700 1 --2- 172.21.15.194:0/4272037319 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00f5830 unknown :-1 s=CLOSED pgs=275 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:08.835 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.842+0000 7f54a8557700 1 -- 172.21.15.194:0/4272037319 >> 172.21.15.194:0/4272037319 conn(0x7f54a00f0b70 msgr2=0x7f54a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.842+0000 7f54a8557700 1 -- 172.21.15.194:0/4272037319 shutdown_connections 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.842+0000 7f54a8557700 1 -- 172.21.15.194:0/4272037319 wait complete. 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.843+0000 7f54a8557700 1 Processor -- start 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.843+0000 7f54a8557700 1 -- start start 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.843+0000 7f54a8557700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.843+0000 7f54a8557700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f54a00fb820 con 0x7f54a00f5410 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.843+0000 7f54a62f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.843+0000 7f54a62f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48162/0 (socket says 172.21.15.194:48162) 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.843+0000 7f54a62f3700 1 -- 172.21.15.194:0/1513718883 learned_addr learned my addr 172.21.15.194:0/1513718883 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.844+0000 7f54a62f3700 1 -- 172.21.15.194:0/1513718883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5490005040 con 0x7f54a00f5410 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.844+0000 7f54a62f3700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00fd3a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f5490009f80 tx=0x7f549000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.844+0000 7f5496ffd700 1 -- 172.21.15.194:0/1513718883 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5490009320 con 0x7f54a00f5410 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.844+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f54a00fb9c0 con 0x7f54a00f5410 2024-01-27T23:30:08.836 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.844+0000 7f5496ffd700 1 -- 172.21.15.194:0/1513718883 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5490013ec0 con 0x7f54a00f5410 2024-01-27T23:30:08.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.844+0000 7f5496ffd700 1 -- 172.21.15.194:0/1513718883 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f549001ece0 con 0x7f54a00f5410 2024-01-27T23:30:08.837 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.844+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f54a00fbe50 con 0x7f54a00f5410 2024-01-27T23:30:08.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.845+0000 7f5496ffd700 1 -- 172.21.15.194:0/1513718883 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5490014420 con 0x7f54a00f5410 2024-01-27T23:30:08.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.845+0000 7f5496ffd700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f548c02ff60 0x7f548c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:08.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.845+0000 7f5496ffd700 1 -- 172.21.15.194:0/1513718883 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5490040d80 con 0x7f54a00f5410 2024-01-27T23:30:08.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.845+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f54a0059100 con 0x7f54a00f5410 2024-01-27T23:30:08.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.845+0000 7f54a5af2700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f548c02ff60 0x7f548c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:08.838 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.845+0000 7f54a5af2700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f548c02ff60 0x7f548c032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f549c0096a0 tx=0x7f549c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:08.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:07.302256+0000 mgr.smithi121.hcppvz (mgr.14162) 146 : cluster [DBG] pgmap v139: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:08.841 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:08.850+0000 7f5496ffd700 1 -- 172.21.15.194:0/1513718883 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f549001c8f0 con 0x7f54a00f5410 2024-01-27T23:30:09.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.055+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f54a00fc550 con 0x7f54a00f5410 2024-01-27T23:30:09.046 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.055+0000 7f5496ffd700 1 -- 172.21.15.194:0/1513718883 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f549001c320 con 0x7f54a00f5410 2024-01-27T23:30:09.047 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:09.047 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:09.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f548c02ff60 msgr2=0x7f548c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:09.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f548c02ff60 0x7f548c032420 secure :-1 s=READY pgs=104 cs=0 l=1 rev1=1 rx=0x7f549c0096a0 tx=0x7f549c006b40).stop 2024-01-27T23:30:09.048 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 msgr2=0x7f54a00fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00fd3a0 secure :-1 s=READY pgs=276 cs=0 l=1 rev1=1 rx=0x7f5490009f80 tx=0x7f549000b040).stop 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 shutdown_connections 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f548c02ff60 0x7f548c032420 unknown :-1 s=CLOSED pgs=104 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 --2- 172.21.15.194:0/1513718883 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f54a00f5410 0x7f54a00fd3a0 unknown :-1 s=CLOSED pgs=276 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 >> 172.21.15.194:0/1513718883 conn(0x7f54a00f0b70 msgr2=0x7f54a005fe40 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 shutdown_connections 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:09.057+0000 7f54a8557700 1 -- 172.21.15.194:0/1513718883 wait complete. 2024-01-27T23:30:09.049 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:09.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:09.055494+0000 mon.smithi121 (mon.0) 219 : audit [DBG] from='client.? 172.21.15.194:0/1513718883' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:10.437 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:10.437 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:10.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:09.302570+0000 mgr.smithi121.hcppvz (mgr.14162) 147 : cluster [DBG] pgmap v140: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:11.576 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.583+0000 7f255d824700 1 -- 172.21.15.194:0/768972674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 msgr2=0x7f25580f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.583+0000 7f255d824700 1 --2- 172.21.15.194:0/768972674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f25580f4be0 secure :-1 s=READY pgs=277 cs=0 l=1 rev1=1 rx=0x7f2548009fb0 tx=0x7f2548009b30).stop 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 -- 172.21.15.194:0/768972674 shutdown_connections 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 --2- 172.21.15.194:0/768972674 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f25580f4be0 unknown :-1 s=CLOSED pgs=277 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 -- 172.21.15.194:0/768972674 >> 172.21.15.194:0/768972674 conn(0x7f25580f02e0 msgr2=0x7f25580f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 -- 172.21.15.194:0/768972674 shutdown_connections 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 -- 172.21.15.194:0/768972674 wait complete. 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 Processor -- start 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 -- start start 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f2558106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.584+0000 7f255d824700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2558106bf0 con 0x7f25580f47c0 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f2556ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f2558106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:11.577 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f2556ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f2558106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48186/0 (socket says 172.21.15.194:48186) 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f2556ffd700 1 -- 172.21.15.194:0/3260893236 learned_addr learned my addr 172.21.15.194:0/3260893236 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f2556ffd700 1 -- 172.21.15.194:0/3260893236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2548005040 con 0x7f25580f47c0 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f2556ffd700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f2558106700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f2548013510 tx=0x7f254800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f253f7fe700 1 -- 172.21.15.194:0/3260893236 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2548009320 con 0x7f25580f47c0 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2558106df0 con 0x7f25580f47c0 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f253f7fe700 1 -- 172.21.15.194:0/3260893236 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2548013f00 con 0x7f25580f47c0 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f253f7fe700 1 -- 172.21.15.194:0/3260893236 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2548014d90 con 0x7f25580f47c0 2024-01-27T23:30:11.578 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.585+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2558107280 con 0x7f25580f47c0 2024-01-27T23:30:11.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.586+0000 7f253f7fe700 1 -- 172.21.15.194:0/3260893236 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f25480263a0 con 0x7f25580f47c0 2024-01-27T23:30:11.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.586+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2558045ad0 con 0x7f25580f47c0 2024-01-27T23:30:11.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.586+0000 7f253f7fe700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f254002ff60 0x7f2540032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:11.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.586+0000 7f253f7fe700 1 -- 172.21.15.194:0/3260893236 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f25480413b0 con 0x7f25580f47c0 2024-01-27T23:30:11.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.587+0000 7f25567fc700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f254002ff60 0x7f2540032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:11.579 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.587+0000 7f25567fc700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f254002ff60 0x7f2540032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f254c0096a0 tx=0x7f254c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:11.582 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.591+0000 7f253f7fe700 1 -- 172.21.15.194:0/3260893236 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f254801ccf0 con 0x7f25580f47c0 2024-01-27T23:30:11.791 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.800+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2558108130 con 0x7f25580f47c0 2024-01-27T23:30:11.792 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.801+0000 7f253f7fe700 1 -- 172.21.15.194:0/3260893236 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f254801cb10 con 0x7f25580f47c0 2024-01-27T23:30:11.792 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:11.792 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f254002ff60 msgr2=0x7f2540032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f254002ff60 0x7f2540032420 secure :-1 s=READY pgs=105 cs=0 l=1 rev1=1 rx=0x7f254c0096a0 tx=0x7f254c006b40).stop 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 msgr2=0x7f2558106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f2558106700 secure :-1 s=READY pgs=278 cs=0 l=1 rev1=1 rx=0x7f2548013510 tx=0x7f254800b040).stop 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 shutdown_connections 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f254002ff60 0x7f2540032420 unknown :-1 s=CLOSED pgs=105 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 --2- 172.21.15.194:0/3260893236 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f25580f47c0 0x7f2558106700 unknown :-1 s=CLOSED pgs=278 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 >> 172.21.15.194:0/3260893236 conn(0x7f25580f02e0 msgr2=0x7f25580f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 shutdown_connections 2024-01-27T23:30:11.794 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:11.802+0000 7f255d824700 1 -- 172.21.15.194:0/3260893236 wait complete. 2024-01-27T23:30:11.795 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:12.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:11.302959+0000 mgr.smithi121.hcppvz (mgr.14162) 148 : cluster [DBG] pgmap v141: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:12.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:11.800875+0000 mon.smithi121 (mon.0) 220 : audit [DBG] from='client.? 172.21.15.194:0/3260893236' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:13.171 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:13.172 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:14.255 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.264+0000 7f846bfff700 1 -- 172.21.15.194:0/2545567671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 msgr2=0x7f846c0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:14.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.264+0000 7f846bfff700 1 --2- 172.21.15.194:0/2545567671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c0f55c0 secure :-1 s=READY pgs=279 cs=0 l=1 rev1=1 rx=0x7f845c009fb0 tx=0x7f845c009b30).stop 2024-01-27T23:30:14.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.264+0000 7f846bfff700 1 -- 172.21.15.194:0/2545567671 shutdown_connections 2024-01-27T23:30:14.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.264+0000 7f846bfff700 1 --2- 172.21.15.194:0/2545567671 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c0f55c0 unknown :-1 s=CLOSED pgs=279 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:14.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.264+0000 7f846bfff700 1 -- 172.21.15.194:0/2545567671 >> 172.21.15.194:0/2545567671 conn(0x7f846c0f09a0 msgr2=0x7f846c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:14.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.264+0000 7f846bfff700 1 -- 172.21.15.194:0/2545567671 shutdown_connections 2024-01-27T23:30:14.256 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.264+0000 7f846bfff700 1 -- 172.21.15.194:0/2545567671 wait complete. 2024-01-27T23:30:14.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.265+0000 7f846bfff700 1 Processor -- start 2024-01-27T23:30:14.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.265+0000 7f846bfff700 1 -- start start 2024-01-27T23:30:14.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.265+0000 7f846bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c1091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:14.257 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.265+0000 7f846bfff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f846c060220 con 0x7f846c0f51a0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.265+0000 7f846affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c1091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.265+0000 7f846affd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c1091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:48208/0 (socket says 172.21.15.194:48208) 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.265+0000 7f846affd700 1 -- 172.21.15.194:0/4039707742 learned_addr learned my addr 172.21.15.194:0/4039707742 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.266+0000 7f846affd700 1 -- 172.21.15.194:0/4039707742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f845c005040 con 0x7f846c0f51a0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.266+0000 7f846affd700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c1091d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f845c013510 tx=0x7f845c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.266+0000 7f84537fe700 1 -- 172.21.15.194:0/4039707742 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f845c009320 con 0x7f846c0f51a0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.266+0000 7f84537fe700 1 -- 172.21.15.194:0/4039707742 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f845c013f00 con 0x7f846c0f51a0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.266+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f846c060420 con 0x7f846c0f51a0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.266+0000 7f84537fe700 1 -- 172.21.15.194:0/4039707742 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f845c014dc0 con 0x7f846c0f51a0 2024-01-27T23:30:14.258 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.266+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f846c109880 con 0x7f846c0f51a0 2024-01-27T23:30:14.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.267+0000 7f84537fe700 1 -- 172.21.15.194:0/4039707742 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f845c014420 con 0x7f846c0f51a0 2024-01-27T23:30:14.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.267+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f846c059100 con 0x7f846c0f51a0 2024-01-27T23:30:14.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.267+0000 7f84537fe700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f845402ff60 0x7f8454032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:14.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.267+0000 7f84537fe700 1 -- 172.21.15.194:0/4039707742 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f845c041110 con 0x7f846c0f51a0 2024-01-27T23:30:14.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.267+0000 7f846a7fc700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f845402ff60 0x7f8454032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:14.259 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.268+0000 7f846a7fc700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f845402ff60 0x7f8454032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f84600096a0 tx=0x7f8460006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:14.263 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.272+0000 7f84537fe700 1 -- 172.21.15.194:0/4039707742 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f845c026bf0 con 0x7f846c0f51a0 2024-01-27T23:30:14.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.481+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f846c045ad0 con 0x7f846c0f51a0 2024-01-27T23:30:14.474 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.482+0000 7f84537fe700 1 -- 172.21.15.194:0/4039707742 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f845c01cb10 con 0x7f846c0f51a0 2024-01-27T23:30:14.474 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:14.474 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:14.475 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f845402ff60 msgr2=0x7f8454032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:14.475 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f845402ff60 0x7f8454032420 secure :-1 s=READY pgs=106 cs=0 l=1 rev1=1 rx=0x7f84600096a0 tx=0x7f8460006b40).stop 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 msgr2=0x7f846c1091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c1091d0 secure :-1 s=READY pgs=280 cs=0 l=1 rev1=1 rx=0x7f845c013510 tx=0x7f845c00b040).stop 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 shutdown_connections 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f845402ff60 0x7f8454032420 unknown :-1 s=CLOSED pgs=106 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 --2- 172.21.15.194:0/4039707742 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f846c0f51a0 0x7f846c1091d0 unknown :-1 s=CLOSED pgs=280 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 >> 172.21.15.194:0/4039707742 conn(0x7f846c0f09a0 msgr2=0x7f846c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 shutdown_connections 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:14.484+0000 7f846bfff700 1 -- 172.21.15.194:0/4039707742 wait complete. 2024-01-27T23:30:14.476 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:14.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:13.303317+0000 mgr.smithi121.hcppvz (mgr.14162) 149 : cluster [DBG] pgmap v142: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:15.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:15 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:14.482511+0000 mon.smithi121 (mon.0) 221 : audit [DBG] from='client.? 172.21.15.194:0/4039707742' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:15.858 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:15.858 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:16.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:15.303637+0000 mgr.smithi121.hcppvz (mgr.14162) 150 : cluster [DBG] pgmap v143: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:17.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 -- 172.21.15.194:0/3026176334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 msgr2=0x7f24600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:17.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 --2- 172.21.15.194:0/3026176334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f24600f5830 secure :-1 s=READY pgs=281 cs=0 l=1 rev1=1 rx=0x7f2454009fb0 tx=0x7f2454009b30).stop 2024-01-27T23:30:17.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 -- 172.21.15.194:0/3026176334 shutdown_connections 2024-01-27T23:30:17.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 --2- 172.21.15.194:0/3026176334 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f24600f5830 unknown :-1 s=CLOSED pgs=281 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:17.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 -- 172.21.15.194:0/3026176334 >> 172.21.15.194:0/3026176334 conn(0x7f24600f0b70 msgr2=0x7f24600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:17.015 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 -- 172.21.15.194:0/3026176334 shutdown_connections 2024-01-27T23:30:17.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 -- 172.21.15.194:0/3026176334 wait complete. 2024-01-27T23:30:17.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 Processor -- start 2024-01-27T23:30:17.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.023+0000 7f2466fef700 1 -- start start 2024-01-27T23:30:17.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.024+0000 7f2466fef700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f2460105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:17.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.024+0000 7f2466fef700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2460105b20 con 0x7f24600f5410 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.024+0000 7f2464d8b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f2460105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.024+0000 7f2464d8b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f2460105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45880/0 (socket says 172.21.15.194:45880) 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.024+0000 7f2464d8b700 1 -- 172.21.15.194:0/1249442998 learned_addr learned my addr 172.21.15.194:0/1249442998 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.025+0000 7f2464d8b700 1 -- 172.21.15.194:0/1249442998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2454005040 con 0x7f24600f5410 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.025+0000 7f2464d8b700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f2460105630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2454013510 tx=0x7f245400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.025+0000 7f245d7fa700 1 -- 172.21.15.194:0/1249442998 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2454009320 con 0x7f24600f5410 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.025+0000 7f245d7fa700 1 -- 172.21.15.194:0/1249442998 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2454013f00 con 0x7f24600f5410 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.025+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2460105d20 con 0x7f24600f5410 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.025+0000 7f245d7fa700 1 -- 172.21.15.194:0/1249442998 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2454014d90 con 0x7f24600f5410 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.025+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f24601061b0 con 0x7f24600f5410 2024-01-27T23:30:17.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.026+0000 7f245d7fa700 1 -- 172.21.15.194:0/1249442998 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f24540263a0 con 0x7f24600f5410 2024-01-27T23:30:17.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.026+0000 7f245d7fa700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f245002ff60 0x7f2450032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:17.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.026+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2460045ad0 con 0x7f24600f5410 2024-01-27T23:30:17.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.026+0000 7f245d7fa700 1 -- 172.21.15.194:0/1249442998 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2454040fd0 con 0x7f24600f5410 2024-01-27T23:30:17.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.026+0000 7f245ffff700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f245002ff60 0x7f2450032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:17.019 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.026+0000 7f245ffff700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f245002ff60 0x7f2450032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f244c0096a0 tx=0x7f244c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:17.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.030+0000 7f245d7fa700 1 -- 172.21.15.194:0/1249442998 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f245406d2b0 con 0x7f24600f5410 2024-01-27T23:30:17.227 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.235+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f24600faf50 con 0x7f24600f5410 2024-01-27T23:30:17.227 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.236+0000 7f245d7fa700 1 -- 172.21.15.194:0/1249442998 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f24540126f0 con 0x7f24600f5410 2024-01-27T23:30:17.228 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:17.228 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:17.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f245002ff60 msgr2=0x7f2450032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:17.229 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f245002ff60 0x7f2450032420 secure :-1 s=READY pgs=107 cs=0 l=1 rev1=1 rx=0x7f244c0096a0 tx=0x7f244c006b40).stop 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 msgr2=0x7f2460105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f2460105630 secure :-1 s=READY pgs=282 cs=0 l=1 rev1=1 rx=0x7f2454013510 tx=0x7f245400b040).stop 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 shutdown_connections 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f245002ff60 0x7f2450032420 unknown :-1 s=CLOSED pgs=107 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 --2- 172.21.15.194:0/1249442998 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f24600f5410 0x7f2460105630 unknown :-1 s=CLOSED pgs=282 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 >> 172.21.15.194:0/1249442998 conn(0x7f24600f0b70 msgr2=0x7f24600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 shutdown_connections 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:17.238+0000 7f2466fef700 1 -- 172.21.15.194:0/1249442998 wait complete. 2024-01-27T23:30:17.230 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:17.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:17 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:17.236569+0000 mon.smithi121 (mon.0) 222 : audit [DBG] from='client.? 172.21.15.194:0/1249442998' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:18.595 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:18.595 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:18.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:17.304011+0000 mgr.smithi121.hcppvz (mgr.14162) 151 : cluster [DBG] pgmap v144: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:18.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:17.396124+0000 mon.smithi121 (mon.0) 223 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:30:18.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:17.404573+0000 mon.smithi121 (mon.0) 224 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:30:19.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 -- 172.21.15.194:0/4256586217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 msgr2=0x7ff6d80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:19.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 --2- 172.21.15.194:0/4256586217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d80f55c0 secure :-1 s=READY pgs=283 cs=0 l=1 rev1=1 rx=0x7ff6cc009fb0 tx=0x7ff6cc009b30).stop 2024-01-27T23:30:19.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 -- 172.21.15.194:0/4256586217 shutdown_connections 2024-01-27T23:30:19.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 --2- 172.21.15.194:0/4256586217 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d80f55c0 unknown :-1 s=CLOSED pgs=283 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 -- 172.21.15.194:0/4256586217 >> 172.21.15.194:0/4256586217 conn(0x7ff6d80f09a0 msgr2=0x7ff6d80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 -- 172.21.15.194:0/4256586217 shutdown_connections 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 -- 172.21.15.194:0/4256586217 wait complete. 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 Processor -- start 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.717+0000 7ff6dd877700 1 -- start start 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.718+0000 7ff6dd877700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d81064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.718+0000 7ff6dd877700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff6d8106990 con 0x7ff6d80f51a0 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.718+0000 7ff6dc875700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d81064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.718+0000 7ff6dc875700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d81064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45902/0 (socket says 172.21.15.194:45902) 2024-01-27T23:30:19.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.718+0000 7ff6dc875700 1 -- 172.21.15.194:0/2391288451 learned_addr learned my addr 172.21.15.194:0/2391288451 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.718+0000 7ff6dc875700 1 -- 172.21.15.194:0/2391288451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff6cc005040 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.718+0000 7ff6dc875700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d81064a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff6cc009830 tx=0x7ff6cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.719+0000 7ff6d57fa700 1 -- 172.21.15.194:0/2391288451 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6cc009320 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.719+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff6d8106b90 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.719+0000 7ff6d57fa700 1 -- 172.21.15.194:0/2391288451 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff6cc0136c0 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.719+0000 7ff6d57fa700 1 -- 172.21.15.194:0/2391288451 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff6cc01ee90 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.719+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff6d8107020 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.719+0000 7ff6d57fa700 1 -- 172.21.15.194:0/2391288451 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff6cc01e420 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.719+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff6d8045ad0 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.720+0000 7ff6d57fa700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff6c802ff10 0x7ff6c80323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.720+0000 7ff6d57fa700 1 -- 172.21.15.194:0/2391288451 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff6cc040fc0 con 0x7ff6d80f51a0 2024-01-27T23:30:19.712 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.720+0000 7ff6d7fff700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff6c802ff10 0x7ff6c80323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:19.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.720+0000 7ff6d7fff700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff6c802ff10 0x7ff6c80323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff6c40096a0 tx=0x7ff6c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:19.715 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.724+0000 7ff6d57fa700 1 -- 172.21.15.194:0/2391288451 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff6cc06d2b0 con 0x7ff6d80f51a0 2024-01-27T23:30:19.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.932+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff6d8107ed0 con 0x7ff6d80f51a0 2024-01-27T23:30:19.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.933+0000 7ff6d57fa700 1 -- 172.21.15.194:0/2391288451 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff6cc0095f0 con 0x7ff6d80f51a0 2024-01-27T23:30:19.924 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:19.924 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff6c802ff10 msgr2=0x7ff6c80323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff6c802ff10 0x7ff6c80323d0 secure :-1 s=READY pgs=108 cs=0 l=1 rev1=1 rx=0x7ff6c40096a0 tx=0x7ff6c4006b40).stop 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 msgr2=0x7ff6d81064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d81064a0 secure :-1 s=READY pgs=284 cs=0 l=1 rev1=1 rx=0x7ff6cc009830 tx=0x7ff6cc00b040).stop 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 shutdown_connections 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff6c802ff10 0x7ff6c80323d0 unknown :-1 s=CLOSED pgs=108 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 --2- 172.21.15.194:0/2391288451 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff6d80f51a0 0x7ff6d81064a0 unknown :-1 s=CLOSED pgs=284 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:19.926 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 >> 172.21.15.194:0/2391288451 conn(0x7ff6d80f09a0 msgr2=0x7ff6d805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:19.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.934+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 shutdown_connections 2024-01-27T23:30:19.927 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:19.935+0000 7ff6dd877700 1 -- 172.21.15.194:0/2391288451 wait complete. 2024-01-27T23:30:19.927 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:20.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:19.304341+0000 mgr.smithi121.hcppvz (mgr.14162) 152 : cluster [DBG] pgmap v145: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:20.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:19.933070+0000 mon.smithi121 (mon.0) 225 : audit [DBG] from='client.? 172.21.15.194:0/2391288451' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:21.306 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:21.307 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.411+0000 7f2962348700 1 -- 172.21.15.194:0/1637823635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 msgr2=0x7f295c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.411+0000 7f2962348700 1 --2- 172.21.15.194:0/1637823635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c0f6930 secure :-1 s=READY pgs=285 cs=0 l=1 rev1=1 rx=0x7f294c009fb0 tx=0x7f294c009b30).stop 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.411+0000 7f2962348700 1 -- 172.21.15.194:0/1637823635 shutdown_connections 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.411+0000 7f2962348700 1 --2- 172.21.15.194:0/1637823635 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c0f6930 unknown :-1 s=CLOSED pgs=285 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.411+0000 7f2962348700 1 -- 172.21.15.194:0/1637823635 >> 172.21.15.194:0/1637823635 conn(0x7f295c0f1c70 msgr2=0x7f295c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.411+0000 7f2962348700 1 -- 172.21.15.194:0/1637823635 shutdown_connections 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.411+0000 7f2962348700 1 -- 172.21.15.194:0/1637823635 wait complete. 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.412+0000 7f2962348700 1 Processor -- start 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.412+0000 7f2962348700 1 -- start start 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.412+0000 7f2962348700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:22.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.412+0000 7f2962348700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f295c069e20 con 0x7f295c0f6510 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.412+0000 7f295bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f295bfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45920/0 (socket says 172.21.15.194:45920) 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f295bfff700 1 -- 172.21.15.194:0/1432822946 learned_addr learned my addr 172.21.15.194:0/1432822946 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f295bfff700 1 -- 172.21.15.194:0/1432822946 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f294c005040 con 0x7f295c0f6510 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f295bfff700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f294c014000 tx=0x7f294c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f2958ff9700 1 -- 172.21.15.194:0/1432822946 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c0095f0 con 0x7f295c0f6510 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f295c069fc0 con 0x7f295c0f6510 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f2958ff9700 1 -- 172.21.15.194:0/1432822946 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f294c007e70 con 0x7f295c0f6510 2024-01-27T23:30:22.406 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f2958ff9700 1 -- 172.21.15.194:0/1432822946 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f294c014ea0 con 0x7f295c0f6510 2024-01-27T23:30:22.407 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.413+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f295c06a450 con 0x7f295c0f6510 2024-01-27T23:30:22.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.414+0000 7f2958ff9700 1 -- 172.21.15.194:0/1432822946 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f294c025070 con 0x7f295c0f6510 2024-01-27T23:30:22.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.414+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f295c059100 con 0x7f295c0f6510 2024-01-27T23:30:22.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.414+0000 7f2958ff9700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f294402ff60 0x7f2944032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:22.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.414+0000 7f2958ff9700 1 -- 172.21.15.194:0/1432822946 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f294c0409e0 con 0x7f295c0f6510 2024-01-27T23:30:22.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.414+0000 7f295b7fe700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f294402ff60 0x7f2944032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:22.408 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.415+0000 7f295b7fe700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f294402ff60 0x7f2944032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f29500096a0 tx=0x7f2950006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:22.410 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.419+0000 7f2958ff9700 1 -- 172.21.15.194:0/1432822946 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f294c00f4e0 con 0x7f295c0f6510 2024-01-27T23:30:22.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:21.304719+0000 mgr.smithi121.hcppvz (mgr.14162) 153 : cluster [DBG] pgmap v146: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:22.621 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.630+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f295c045ad0 con 0x7f295c0f6510 2024-01-27T23:30:22.622 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.630+0000 7f2958ff9700 1 -- 172.21.15.194:0/1432822946 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f294c00f300 con 0x7f295c0f6510 2024-01-27T23:30:22.622 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:22.622 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f294402ff60 msgr2=0x7f2944032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f294402ff60 0x7f2944032420 secure :-1 s=READY pgs=109 cs=0 l=1 rev1=1 rx=0x7f29500096a0 tx=0x7f2950006b40).stop 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 msgr2=0x7f295c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c06b9a0 secure :-1 s=READY pgs=286 cs=0 l=1 rev1=1 rx=0x7f294c014000 tx=0x7f294c00b040).stop 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 shutdown_connections 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f294402ff60 0x7f2944032420 unknown :-1 s=CLOSED pgs=109 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 --2- 172.21.15.194:0/1432822946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f295c0f6510 0x7f295c06b9a0 unknown :-1 s=CLOSED pgs=286 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:22.624 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 >> 172.21.15.194:0/1432822946 conn(0x7f295c0f1c70 msgr2=0x7f295c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:22.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 shutdown_connections 2024-01-27T23:30:22.625 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:22.632+0000 7f2962348700 1 -- 172.21.15.194:0/1432822946 wait complete. 2024-01-27T23:30:22.625 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:23.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:23 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:22.630666+0000 mon.smithi121 (mon.0) 226 : audit [DBG] from='client.? 172.21.15.194:0/1432822946' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:24.004 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:24.005 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:24.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:23.304987+0000 mgr.smithi121.hcppvz (mgr.14162) 154 : cluster [DBG] pgmap v147: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:25.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.162+0000 7f59cbfff700 1 -- 172.21.15.194:0/989189301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 msgr2=0x7f59cc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.162+0000 7f59cbfff700 1 --2- 172.21.15.194:0/989189301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc0f66c0 secure :-1 s=READY pgs=287 cs=0 l=1 rev1=1 rx=0x7f59c0009fb0 tx=0x7f59c0009b30).stop 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.162+0000 7f59cbfff700 1 -- 172.21.15.194:0/989189301 shutdown_connections 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.162+0000 7f59cbfff700 1 --2- 172.21.15.194:0/989189301 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc0f66c0 unknown :-1 s=CLOSED pgs=287 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.162+0000 7f59cbfff700 1 -- 172.21.15.194:0/989189301 >> 172.21.15.194:0/989189301 conn(0x7f59cc0f1aa0 msgr2=0x7f59cc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.162+0000 7f59cbfff700 1 -- 172.21.15.194:0/989189301 shutdown_connections 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.162+0000 7f59cbfff700 1 -- 172.21.15.194:0/989189301 wait complete. 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59cbfff700 1 Processor -- start 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59cbfff700 1 -- start start 2024-01-27T23:30:25.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59cbfff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59cbfff700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f59cc1069a0 con 0x7f59cc0f62a0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59caffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45940/0 (socket says 172.21.15.194:45940) 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59caffd700 1 -- 172.21.15.194:0/1090081034 learned_addr learned my addr 172.21.15.194:0/1090081034 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59caffd700 1 -- 172.21.15.194:0/1090081034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f59c0005040 con 0x7f59cc0f62a0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59caffd700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc1064b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f59c0009830 tx=0x7f59c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.163+0000 7f59b37fe700 1 -- 172.21.15.194:0/1090081034 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59c0009320 con 0x7f59cc0f62a0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59b37fe700 1 -- 172.21.15.194:0/1090081034 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f59c00136c0 con 0x7f59cc0f62a0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f59cc106ba0 con 0x7f59cc0f62a0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59b37fe700 1 -- 172.21.15.194:0/1090081034 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f59c001ee90 con 0x7f59cc0f62a0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f59cc107030 con 0x7f59cc0f62a0 2024-01-27T23:30:25.157 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59b37fe700 1 -- 172.21.15.194:0/1090081034 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f59c0014440 con 0x7f59cc0f62a0 2024-01-27T23:30:25.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59b37fe700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f59b402ff10 0x7f59b40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:25.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f59cc045ad0 con 0x7f59cc0f62a0 2024-01-27T23:30:25.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.164+0000 7f59b37fe700 1 -- 172.21.15.194:0/1090081034 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f59c0041360 con 0x7f59cc0f62a0 2024-01-27T23:30:25.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.165+0000 7f59ca7fc700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f59b402ff10 0x7f59b40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:25.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.165+0000 7f59ca7fc700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f59b402ff10 0x7f59b40323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f59bc0096a0 tx=0x7f59bc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:25.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.169+0000 7f59b37fe700 1 -- 172.21.15.194:0/1090081034 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f59c00273f0 con 0x7f59cc0f62a0 2024-01-27T23:30:25.363 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.372+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f59cc107ee0 con 0x7f59cc0f62a0 2024-01-27T23:30:25.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.373+0000 7f59b37fe700 1 -- 172.21.15.194:0/1090081034 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f59c002c2d0 con 0x7f59cc0f62a0 2024-01-27T23:30:25.364 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:25.364 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:25.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f59b402ff10 msgr2=0x7f59b40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:25.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f59b402ff10 0x7f59b40323d0 secure :-1 s=READY pgs=110 cs=0 l=1 rev1=1 rx=0x7f59bc0096a0 tx=0x7f59bc006b40).stop 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 msgr2=0x7f59cc1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc1064b0 secure :-1 s=READY pgs=288 cs=0 l=1 rev1=1 rx=0x7f59c0009830 tx=0x7f59c000b040).stop 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 shutdown_connections 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f59b402ff10 0x7f59b40323d0 unknown :-1 s=CLOSED pgs=110 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 --2- 172.21.15.194:0/1090081034 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f59cc0f62a0 0x7f59cc1064b0 unknown :-1 s=CLOSED pgs=288 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 >> 172.21.15.194:0/1090081034 conn(0x7f59cc0f1aa0 msgr2=0x7f59cc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 shutdown_connections 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:25.374+0000 7f59cbfff700 1 -- 172.21.15.194:0/1090081034 wait complete. 2024-01-27T23:30:25.367 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:30:24.669114+0000 mgr.smithi121.hcppvz (mgr.14162) 155 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi121 ... 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:26:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:26:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:27:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:28:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:29:09Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.559 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:29:47Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Traceback (most recent call last): 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: main() 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: r = ctx.func(ctx) 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: return func(ctx) 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-27T23:30:25.560 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: raise RuntimeError('uid/gid not found') 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: RuntimeError: uid/gid not found 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Traceback (most recent call last): 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: yield (conn, connr) 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: code, '\n'.join(err))) 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon alertmanager.smithi121 ... 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/alertmanager 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:26:02Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.561 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:26:39Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:27:17Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Non-zero exit code 125 from /bin/podman run --rm --ipc=host --no-hosts --net=host --entrypoint stat --init -e CONTAINER_IMAGE=docker.io/prom/alertmanager:v0.20.0 -e NODE_NAME=smithi121 -e CEPH_USE_RANDOM_NONCE=1 docker.io/prom/alertmanager:v0.20.0 -c %u %g /etc/prometheus 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Trying to pull docker.io/prom/alertmanager:v0.20.0... 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:28:31Z" level=warning msg="Failed, retrying in 1s ... (1/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:29:09Z" level=warning msg="Failed, retrying in 1s ... (2/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr time="2024-01-27T23:29:47Z" level=warning msg="Failed, retrying in 1s ... (3/3). Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get \"https://docker-mirror.front.sepia.ceph.com:5000/v2/\": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit" 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: stat: stderr Error: initializing source docker://prom/alertmanager:v0.20.0: (Mirrors also failed: [docker-mirror.front.sepia.ceph.com:5000/prom/alertmanager:v0.20.0: pinging container registry docker-mirror.front.sepia.ceph.com:5000: Get "https://docker-mirror.front.sepia.ceph.com:5000/v2/": dial tcp 172.21.0.79:5000: connect: no route to host]): docker.io/prom/alertmanager:v0.20.0: reading manifest v0.20.0 in docker.io/prom/alertmanager: toomanyrequests: You have reached your pull rate limit. You may increase the limit by authenticating and upgrading: https://www.docker.com/increase-rate-limit 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Traceback (most recent call last): 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: main() 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: r = ctx.func(ctx) 2024-01-27T23:30:25.562 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: return func(ctx) 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4263, in command_deploy 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: uid, gid = extract_uid_gid_monitoring(ctx, daemon_type) 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4191, in extract_uid_gid_monitoring 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: uid, gid = extract_uid_gid(ctx, file_path=['/etc/alertmanager', '/etc/prometheus']) 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2531, in extract_uid_gid 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: raise RuntimeError('uid/gid not found') 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: RuntimeError: uid/gid not found 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:24.670468+0000 mon.smithi121 (mon.0) 227 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]: dispatch 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:24.672864+0000 mon.smithi121 (mon.0) 228 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd='[{"prefix": "auth get-or-create", "entity": "client.crash.smithi121", "caps": ["mon", "profile crash", "mgr", "profile crash"]}]': finished 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:24.673681+0000 mon.smithi121 (mon.0) 229 : audit [DBG] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "config generate-minimal-conf"}]: dispatch 2024-01-27T23:30:25.563 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:30:24.674444+0000 mgr.smithi121.hcppvz (mgr.14162) 156 : cephadm [INF] Deploying daemon crash.smithi121 on smithi121 2024-01-27T23:30:26.589 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:25.305208+0000 mgr.smithi121.hcppvz (mgr.14162) 157 : cluster [DBG] pgmap v148: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:25.373012+0000 mon.smithi121 (mon.0) 230 : audit [DBG] from='client.? 172.21.15.194:0/1090081034' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:26.218240+0000 mon.smithi121 (mon.0) 231 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:30:26.789 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:26.789 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:27.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:26.345853+0000 mon.smithi121 (mon.0) 232 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:30:27.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:26.347648+0000 mon.smithi121 (mon.0) 233 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:30:27.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:26.348069+0000 mon.smithi121 (mon.0) 234 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-27T23:30:27.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:26.348469+0000 mgr.smithi121.hcppvz (mgr.14162) 158 : audit [DBG] from='mon.0 -' entity='mon.' cmd=[{"prefix": "dashboard set-grafana-api-ssl-verify", "value": "false"}]: dispatch 2024-01-27T23:30:27.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:26.350662+0000 mon.smithi121 (mon.0) 235 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:30:27.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:30:26.353369+0000 mgr.smithi121.hcppvz (mgr.14162) 159 : cephadm [INF] Deploying daemon grafana.smithi121 on smithi121 2024-01-27T23:30:27.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/1594620282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 msgr2=0x7f8cec0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:27.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 --2- 172.21.15.194:0/1594620282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec0f6930 secure :-1 s=READY pgs=289 cs=0 l=1 rev1=1 rx=0x7f8ce0009fb0 tx=0x7f8ce0009b30).stop 2024-01-27T23:30:27.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/1594620282 shutdown_connections 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 --2- 172.21.15.194:0/1594620282 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec0f6930 unknown :-1 s=CLOSED pgs=289 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/1594620282 >> 172.21.15.194:0/1594620282 conn(0x7f8cec0f1c70 msgr2=0x7f8cec0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/1594620282 shutdown_connections 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/1594620282 wait complete. 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.882+0000 7f8cf2bd0700 1 Processor -- start 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf2bd0700 1 -- start start 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf2bd0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf2bd0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8cec106c20 con 0x7f8cec0f6510 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf096c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf096c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:38722/0 (socket says 172.21.15.194:38722) 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf096c700 1 -- 172.21.15.194:0/3013545598 learned_addr learned my addr 172.21.15.194:0/3013545598 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf096c700 1 -- 172.21.15.194:0/3013545598 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8ce0005040 con 0x7f8cec0f6510 2024-01-27T23:30:27.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.883+0000 7f8cf096c700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec106730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f8ce0013510 tx=0x7f8ce000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.884+0000 7f8ce97fa700 1 -- 172.21.15.194:0/3013545598 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce0009320 con 0x7f8cec0f6510 2024-01-27T23:30:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.884+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8cec106e20 con 0x7f8cec0f6510 2024-01-27T23:30:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.884+0000 7f8ce97fa700 1 -- 172.21.15.194:0/3013545598 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8ce0013f00 con 0x7f8cec0f6510 2024-01-27T23:30:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.884+0000 7f8ce97fa700 1 -- 172.21.15.194:0/3013545598 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8ce0014d90 con 0x7f8cec0f6510 2024-01-27T23:30:27.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.884+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8cec1072b0 con 0x7f8cec0f6510 2024-01-27T23:30:27.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.884+0000 7f8ce97fa700 1 -- 172.21.15.194:0/3013545598 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8ce0014420 con 0x7f8cec0f6510 2024-01-27T23:30:27.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.885+0000 7f8ce97fa700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cdc02ffb0 0x7f8cdc032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:27.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.885+0000 7f8ce97fa700 1 -- 172.21.15.194:0/3013545598 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8ce00410e0 con 0x7f8cec0f6510 2024-01-27T23:30:27.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.885+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8cec059100 con 0x7f8cec0f6510 2024-01-27T23:30:27.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.885+0000 7f8cebfff700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cdc02ffb0 0x7f8cdc032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:27.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.885+0000 7f8cebfff700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cdc02ffb0 0x7f8cdc032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8cd80096a0 tx=0x7f8cd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:27.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:27.890+0000 7f8ce97fa700 1 -- 172.21.15.194:0/3013545598 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8ce00095f0 con 0x7f8cec0f6510 2024-01-27T23:30:28.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.099+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8cec045ad0 con 0x7f8cec0f6510 2024-01-27T23:30:28.091 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.100+0000 7f8ce97fa700 1 -- 172.21.15.194:0/3013545598 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8ce002b020 con 0x7f8cec0f6510 2024-01-27T23:30:28.091 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:28.091 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:28.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cdc02ffb0 msgr2=0x7f8cdc032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:28.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cdc02ffb0 0x7f8cdc032470 secure :-1 s=READY pgs=111 cs=0 l=1 rev1=1 rx=0x7f8cd80096a0 tx=0x7f8cd8006b40).stop 2024-01-27T23:30:28.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 msgr2=0x7f8cec106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:28.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec106730 secure :-1 s=READY pgs=290 cs=0 l=1 rev1=1 rx=0x7f8ce0013510 tx=0x7f8ce000b040).stop 2024-01-27T23:30:28.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 shutdown_connections 2024-01-27T23:30:28.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8cdc02ffb0 0x7f8cdc032470 unknown :-1 s=CLOSED pgs=111 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:28.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 --2- 172.21.15.194:0/3013545598 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8cec0f6510 0x7f8cec106730 unknown :-1 s=CLOSED pgs=290 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:28.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 >> 172.21.15.194:0/3013545598 conn(0x7f8cec0f1c70 msgr2=0x7f8cec0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:28.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 shutdown_connections 2024-01-27T23:30:28.094 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:28.101+0000 7f8cf2bd0700 1 -- 172.21.15.194:0/3013545598 wait complete. 2024-01-27T23:30:28.094 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:28.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:27.305517+0000 mgr.smithi121.hcppvz (mgr.14162) 160 : cluster [DBG] pgmap v149: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:28.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:28.099901+0000 mon.smithi121 (mon.0) 236 : audit [DBG] from='client.? 172.21.15.194:0/3013545598' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:29.483 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:29.483 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:30.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.686+0000 7f284482a700 1 -- 172.21.15.194:0/255153918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 msgr2=0x7f28400f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:30.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.686+0000 7f284482a700 1 --2- 172.21.15.194:0/255153918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f28400f6930 secure :-1 s=READY pgs=291 cs=0 l=1 rev1=1 rx=0x7f2834009fb0 tx=0x7f2834009b30).stop 2024-01-27T23:30:30.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.686+0000 7f284482a700 1 -- 172.21.15.194:0/255153918 shutdown_connections 2024-01-27T23:30:30.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.686+0000 7f284482a700 1 --2- 172.21.15.194:0/255153918 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f28400f6930 unknown :-1 s=CLOSED pgs=291 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:30.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.686+0000 7f284482a700 1 -- 172.21.15.194:0/255153918 >> 172.21.15.194:0/255153918 conn(0x7f28400f1c70 msgr2=0x7f28400f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:30.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.686+0000 7f284482a700 1 -- 172.21.15.194:0/255153918 shutdown_connections 2024-01-27T23:30:30.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.686+0000 7f284482a700 1 -- 172.21.15.194:0/255153918 wait complete. 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.687+0000 7f284482a700 1 Processor -- start 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.687+0000 7f284482a700 1 -- start start 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.687+0000 7f284482a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f2840106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.687+0000 7f284482a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2840106c20 con 0x7f28400f6510 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.687+0000 7f283e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f2840106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.687+0000 7f283e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f2840106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:38742/0 (socket says 172.21.15.194:38742) 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.687+0000 7f283e59c700 1 -- 172.21.15.194:0/3525499169 learned_addr learned my addr 172.21.15.194:0/3525499169 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.688+0000 7f283e59c700 1 -- 172.21.15.194:0/3525499169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2834005040 con 0x7f28400f6510 2024-01-27T23:30:30.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.688+0000 7f283e59c700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f2840106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2834013510 tx=0x7f283400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.688+0000 7f282effd700 1 -- 172.21.15.194:0/3525499169 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2834009320 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.688+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2840106e20 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.688+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f28401072b0 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.688+0000 7f282effd700 1 -- 172.21.15.194:0/3525499169 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2834013f00 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.688+0000 7f282effd700 1 -- 172.21.15.194:0/3525499169 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2834014d60 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.689+0000 7f282effd700 1 -- 172.21.15.194:0/3525499169 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2834014420 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.689+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2840045ad0 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.689+0000 7f282effd700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f282802ff60 0x7f2828032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.689+0000 7f282effd700 1 -- 172.21.15.194:0/3525499169 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f28340412c0 con 0x7f28400f6510 2024-01-27T23:30:30.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.689+0000 7f283dd9b700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f282802ff60 0x7f2828032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:30.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.690+0000 7f283dd9b700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f282802ff60 0x7f2828032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f28300096a0 tx=0x7f2830006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:30.685 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.694+0000 7f282effd700 1 -- 172.21.15.194:0/3525499169 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2834012d00 con 0x7f28400f6510 2024-01-27T23:30:30.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:29.305853+0000 mgr.smithi121.hcppvz (mgr.14162) 161 : cluster [DBG] pgmap v150: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:30.892 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.901+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f28400fbf60 con 0x7f28400f6510 2024-01-27T23:30:30.893 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.901+0000 7f282effd700 1 -- 172.21.15.194:0/3525499169 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2834012b20 con 0x7f28400f6510 2024-01-27T23:30:30.893 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:30.893 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f282802ff60 msgr2=0x7f2828032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f282802ff60 0x7f2828032420 secure :-1 s=READY pgs=112 cs=0 l=1 rev1=1 rx=0x7f28300096a0 tx=0x7f2830006b40).stop 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 msgr2=0x7f2840106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f2840106730 secure :-1 s=READY pgs=292 cs=0 l=1 rev1=1 rx=0x7f2834013510 tx=0x7f283400b040).stop 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 shutdown_connections 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f282802ff60 0x7f2828032420 unknown :-1 s=CLOSED pgs=112 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 --2- 172.21.15.194:0/3525499169 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f28400f6510 0x7f2840106730 unknown :-1 s=CLOSED pgs=292 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 >> 172.21.15.194:0/3525499169 conn(0x7f28400f1c70 msgr2=0x7f28400f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:30.895 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 shutdown_connections 2024-01-27T23:30:30.896 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:30.903+0000 7f284482a700 1 -- 172.21.15.194:0/3525499169 wait complete. 2024-01-27T23:30:30.896 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:31.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:31 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:30.901690+0000 mon.smithi121 (mon.0) 237 : audit [DBG] from='client.? 172.21.15.194:0/3525499169' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:32.288 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:32.289 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:32.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:31.306215+0000 mgr.smithi121.hcppvz (mgr.14162) 162 : cluster [DBG] pgmap v151: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.419+0000 7fccb6949700 1 -- 172.21.15.194:0/2157190734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 msgr2=0x7fccb00f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.419+0000 7fccb6949700 1 --2- 172.21.15.194:0/2157190734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f90f0 secure :-1 s=READY pgs=293 cs=0 l=1 rev1=1 rx=0x7fcca0009fb0 tx=0x7fcca0009b30).stop 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.419+0000 7fccb6949700 1 -- 172.21.15.194:0/2157190734 shutdown_connections 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.419+0000 7fccb6949700 1 --2- 172.21.15.194:0/2157190734 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f90f0 unknown :-1 s=CLOSED pgs=293 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.419+0000 7fccb6949700 1 -- 172.21.15.194:0/2157190734 >> 172.21.15.194:0/2157190734 conn(0x7fccb00f0b70 msgr2=0x7fccb00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.419+0000 7fccb6949700 1 -- 172.21.15.194:0/2157190734 shutdown_connections 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.419+0000 7fccb6949700 1 -- 172.21.15.194:0/2157190734 wait complete. 2024-01-27T23:30:33.413 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.420+0000 7fccb6949700 1 Processor -- start 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.420+0000 7fccb6949700 1 -- start start 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.420+0000 7fccb6949700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.420+0000 7fccb6949700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fccb00f4ff0 con 0x7fccb0060280 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.420+0000 7fccaffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.420+0000 7fccaffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:38752/0 (socket says 172.21.15.194:38752) 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.420+0000 7fccaffff700 1 -- 172.21.15.194:0/2718994312 learned_addr learned my addr 172.21.15.194:0/2718994312 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.421+0000 7fccaffff700 1 -- 172.21.15.194:0/2718994312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcca0005040 con 0x7fccb0060280 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.421+0000 7fccaffff700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f6b70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcca0013510 tx=0x7fcca000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.421+0000 7fccacff9700 1 -- 172.21.15.194:0/2718994312 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca0009320 con 0x7fccb0060280 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.421+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fccb00f5190 con 0x7fccb0060280 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.421+0000 7fccacff9700 1 -- 172.21.15.194:0/2718994312 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcca0013f00 con 0x7fccb0060280 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.421+0000 7fccacff9700 1 -- 172.21.15.194:0/2718994312 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcca0014dc0 con 0x7fccb0060280 2024-01-27T23:30:33.414 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.421+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fccb00f5620 con 0x7fccb0060280 2024-01-27T23:30:33.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.422+0000 7fccacff9700 1 -- 172.21.15.194:0/2718994312 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcca0014420 con 0x7fccb0060280 2024-01-27T23:30:33.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.422+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fccb0059100 con 0x7fccb0060280 2024-01-27T23:30:33.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.422+0000 7fccacff9700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc9802ff60 0x7fcc98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:33.415 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.422+0000 7fccacff9700 1 -- 172.21.15.194:0/2718994312 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcca00410f0 con 0x7fccb0060280 2024-01-27T23:30:33.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.422+0000 7fccaf7fe700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc9802ff60 0x7fcc98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:33.416 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.422+0000 7fccaf7fe700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc9802ff60 0x7fcc98032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcca40096a0 tx=0x7fcca4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:33.418 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.427+0000 7fccacff9700 1 -- 172.21.15.194:0/2718994312 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcca00095f0 con 0x7fccb0060280 2024-01-27T23:30:33.626 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.634+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fccb00fbf50 con 0x7fccb0060280 2024-01-27T23:30:33.627 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.635+0000 7fccacff9700 1 -- 172.21.15.194:0/2718994312 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcca001cb10 con 0x7fccb0060280 2024-01-27T23:30:33.627 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:33.627 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc9802ff60 msgr2=0x7fcc98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc9802ff60 0x7fcc98032420 secure :-1 s=READY pgs=113 cs=0 l=1 rev1=1 rx=0x7fcca40096a0 tx=0x7fcca4006b40).stop 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 msgr2=0x7fccb00f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f6b70 secure :-1 s=READY pgs=294 cs=0 l=1 rev1=1 rx=0x7fcca0013510 tx=0x7fcca000b040).stop 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 shutdown_connections 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcc9802ff60 0x7fcc98032420 unknown :-1 s=CLOSED pgs=113 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 --2- 172.21.15.194:0/2718994312 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fccb0060280 0x7fccb00f6b70 unknown :-1 s=CLOSED pgs=294 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 >> 172.21.15.194:0/2718994312 conn(0x7fccb00f0b70 msgr2=0x7fccb00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 shutdown_connections 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:33.637+0000 7fccb6949700 1 -- 172.21.15.194:0/2718994312 wait complete. 2024-01-27T23:30:33.629 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:34.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:33.306562+0000 mgr.smithi121.hcppvz (mgr.14162) 163 : cluster [DBG] pgmap v152: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:34.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:33.635564+0000 mon.smithi121 (mon.0) 238 : audit [DBG] from='client.? 172.21.15.194:0/2718994312' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:35.013 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:35.013 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:36.103 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.110+0000 7f67d0625700 1 -- 172.21.15.194:0/2094042848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 msgr2=0x7f67c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.110+0000 7f67d0625700 1 --2- 172.21.15.194:0/2094042848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c80f5830 secure :-1 s=READY pgs=295 cs=0 l=1 rev1=1 rx=0x7f67b8009fb0 tx=0x7f67b8009b30).stop 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.110+0000 7f67d0625700 1 -- 172.21.15.194:0/2094042848 shutdown_connections 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.110+0000 7f67d0625700 1 --2- 172.21.15.194:0/2094042848 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c80f5830 unknown :-1 s=CLOSED pgs=295 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.110+0000 7f67d0625700 1 -- 172.21.15.194:0/2094042848 >> 172.21.15.194:0/2094042848 conn(0x7f67c80f0b70 msgr2=0x7f67c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.110+0000 7f67d0625700 1 -- 172.21.15.194:0/2094042848 shutdown_connections 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.110+0000 7f67d0625700 1 -- 172.21.15.194:0/2094042848 wait complete. 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67d0625700 1 Processor -- start 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67d0625700 1 -- start start 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67d0625700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c81055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67d0625700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f67c8105ac0 con 0x7f67c80f5410 2024-01-27T23:30:36.104 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67ce3c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c81055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67ce3c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c81055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40368/0 (socket says 172.21.15.194:40368) 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67ce3c1700 1 -- 172.21.15.194:0/638214464 learned_addr learned my addr 172.21.15.194:0/638214464 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.111+0000 7f67ce3c1700 1 -- 172.21.15.194:0/638214464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f67b8005040 con 0x7f67c80f5410 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.112+0000 7f67ce3c1700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c81055d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f67b8014000 tx=0x7f67b800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.112+0000 7f67beffd700 1 -- 172.21.15.194:0/638214464 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b80095f0 con 0x7f67c80f5410 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.112+0000 7f67beffd700 1 -- 172.21.15.194:0/638214464 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f67b8007e70 con 0x7f67c80f5410 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.112+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f67c8105cc0 con 0x7f67c80f5410 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.112+0000 7f67beffd700 1 -- 172.21.15.194:0/638214464 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67b8014ea0 con 0x7f67c80f5410 2024-01-27T23:30:36.105 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.112+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f67c8106150 con 0x7f67c80f5410 2024-01-27T23:30:36.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.113+0000 7f67beffd700 1 -- 172.21.15.194:0/638214464 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f67b8025070 con 0x7f67c80f5410 2024-01-27T23:30:36.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.113+0000 7f67beffd700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f67b402ff60 0x7f67b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:36.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.113+0000 7f67beffd700 1 -- 172.21.15.194:0/638214464 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f67b8040970 con 0x7f67c80f5410 2024-01-27T23:30:36.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.113+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f67c8045ad0 con 0x7f67c80f5410 2024-01-27T23:30:36.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.113+0000 7f67cdbc0700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f67b402ff60 0x7f67b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:36.106 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.113+0000 7f67cdbc0700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f67b402ff60 0x7f67b4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67c40096a0 tx=0x7f67c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:36.109 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.117+0000 7f67beffd700 1 -- 172.21.15.194:0/638214464 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f67b800f4e0 con 0x7f67c80f5410 2024-01-27T23:30:36.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.323+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f67c8106ea0 con 0x7f67c80f5410 2024-01-27T23:30:36.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.323+0000 7f67beffd700 1 -- 172.21.15.194:0/638214464 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f67b800f300 con 0x7f67c80f5410 2024-01-27T23:30:36.316 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:36.316 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f67b402ff60 msgr2=0x7f67b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f67b402ff60 0x7f67b4032420 secure :-1 s=READY pgs=114 cs=0 l=1 rev1=1 rx=0x7f67c40096a0 tx=0x7f67c4006b40).stop 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 msgr2=0x7f67c81055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c81055d0 secure :-1 s=READY pgs=296 cs=0 l=1 rev1=1 rx=0x7f67b8014000 tx=0x7f67b800b040).stop 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 shutdown_connections 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f67b402ff60 0x7f67b4032420 unknown :-1 s=CLOSED pgs=114 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 --2- 172.21.15.194:0/638214464 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67c80f5410 0x7f67c81055d0 unknown :-1 s=CLOSED pgs=296 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 >> 172.21.15.194:0/638214464 conn(0x7f67c80f0b70 msgr2=0x7f67c80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 shutdown_connections 2024-01-27T23:30:36.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:36.325+0000 7f67d0625700 1 -- 172.21.15.194:0/638214464 wait complete. 2024-01-27T23:30:36.318 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:36.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:35.306871+0000 mgr.smithi121.hcppvz (mgr.14162) 164 : cluster [DBG] pgmap v153: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:36.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:36.323777+0000 mon.smithi121 (mon.0) 239 : audit [DBG] from='client.? 172.21.15.194:0/638214464' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:37.700 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:37.700 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:38.839 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:37.307142+0000 mgr.smithi121.hcppvz (mgr.14162) 165 : cluster [DBG] pgmap v154: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.870+0000 7fa850df0700 1 -- 172.21.15.194:0/3442620040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 msgr2=0x7fa84c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.870+0000 7fa850df0700 1 --2- 172.21.15.194:0/3442620040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c0f6930 secure :-1 s=READY pgs=297 cs=0 l=1 rev1=1 rx=0x7fa834009fb0 tx=0x7fa834009b30).stop 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.870+0000 7fa850df0700 1 -- 172.21.15.194:0/3442620040 shutdown_connections 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.870+0000 7fa850df0700 1 --2- 172.21.15.194:0/3442620040 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c0f6930 unknown :-1 s=CLOSED pgs=297 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.870+0000 7fa850df0700 1 -- 172.21.15.194:0/3442620040 >> 172.21.15.194:0/3442620040 conn(0x7fa84c0f1c70 msgr2=0x7fa84c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.870+0000 7fa850df0700 1 -- 172.21.15.194:0/3442620040 shutdown_connections 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.870+0000 7fa850df0700 1 -- 172.21.15.194:0/3442620040 wait complete. 2024-01-27T23:30:38.865 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa850df0700 1 Processor -- start 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa850df0700 1 -- start start 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa850df0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa850df0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa84c069e20 con 0x7fa84c0f6510 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa84a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa84a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40388/0 (socket says 172.21.15.194:40388) 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa84a59c700 1 -- 172.21.15.194:0/2020360490 learned_addr learned my addr 172.21.15.194:0/2020360490 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa84a59c700 1 -- 172.21.15.194:0/2020360490 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa834005040 con 0x7fa84c0f6510 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa84a59c700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c06b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fa834013510 tx=0x7fa83400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa842ffd700 1 -- 172.21.15.194:0/2020360490 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa834009320 con 0x7fa84c0f6510 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.871+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa84c069fc0 con 0x7fa84c0f6510 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.872+0000 7fa842ffd700 1 -- 172.21.15.194:0/2020360490 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa834013f00 con 0x7fa84c0f6510 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.872+0000 7fa842ffd700 1 -- 172.21.15.194:0/2020360490 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa834014d90 con 0x7fa84c0f6510 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.872+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa84c06a450 con 0x7fa84c0f6510 2024-01-27T23:30:38.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.872+0000 7fa842ffd700 1 -- 172.21.15.194:0/2020360490 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa834014420 con 0x7fa84c0f6510 2024-01-27T23:30:38.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.872+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa84c059100 con 0x7fa84c0f6510 2024-01-27T23:30:38.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.873+0000 7fa842ffd700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa83802ff60 0x7fa838032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:38.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.873+0000 7fa842ffd700 1 -- 172.21.15.194:0/2020360490 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa834040fb0 con 0x7fa84c0f6510 2024-01-27T23:30:38.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.873+0000 7fa849d9b700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa83802ff60 0x7fa838032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:38.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.873+0000 7fa849d9b700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa83802ff60 0x7fa838032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fa83c0096a0 tx=0x7fa83c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:38.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:38.881+0000 7fa842ffd700 1 -- 172.21.15.194:0/2020360490 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa8340095f0 con 0x7fa84c0f6510 2024-01-27T23:30:39.076 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.084+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa84c0fbf60 con 0x7fa84c0f6510 2024-01-27T23:30:39.119 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.085+0000 7fa842ffd700 1 -- 172.21.15.194:0/2020360490 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa83401db70 con 0x7fa84c0f6510 2024-01-27T23:30:39.119 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:39.119 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.086+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa83802ff60 msgr2=0x7fa838032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.086+0000 7fa850df0700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa83802ff60 0x7fa838032420 secure :-1 s=READY pgs=115 cs=0 l=1 rev1=1 rx=0x7fa83c0096a0 tx=0x7fa83c006b40).stop 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.086+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 msgr2=0x7fa84c06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.086+0000 7fa850df0700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c06b9a0 secure :-1 s=READY pgs=298 cs=0 l=1 rev1=1 rx=0x7fa834013510 tx=0x7fa83400b040).stop 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.086+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 shutdown_connections 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.086+0000 7fa850df0700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa83802ff60 0x7fa838032420 unknown :-1 s=CLOSED pgs=115 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.086+0000 7fa850df0700 1 --2- 172.21.15.194:0/2020360490 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa84c0f6510 0x7fa84c06b9a0 unknown :-1 s=CLOSED pgs=298 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:39.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.087+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 >> 172.21.15.194:0/2020360490 conn(0x7fa84c0f1c70 msgr2=0x7fa84c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:39.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.087+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 shutdown_connections 2024-01-27T23:30:39.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:39.087+0000 7fa850df0700 1 -- 172.21.15.194:0/2020360490 wait complete. 2024-01-27T23:30:39.121 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:39.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:39 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:39.085299+0000 mon.smithi121 (mon.0) 240 : audit [DBG] from='client.? 172.21.15.194:0/2020360490' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:40.478 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:40.478 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:40.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:39.307409+0000 mgr.smithi121.hcppvz (mgr.14162) 166 : cluster [DBG] pgmap v155: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:41.596 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.604+0000 7f8ea9d07700 1 -- 172.21.15.194:0/4255281220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 msgr2=0x7f8ea40f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:41.596 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.604+0000 7f8ea9d07700 1 --2- 172.21.15.194:0/4255281220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f90f0 secure :-1 s=READY pgs=299 cs=0 l=1 rev1=1 rx=0x7f8e98009fb0 tx=0x7f8e98009b30).stop 2024-01-27T23:30:41.596 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.604+0000 7f8ea9d07700 1 -- 172.21.15.194:0/4255281220 shutdown_connections 2024-01-27T23:30:41.596 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.604+0000 7f8ea9d07700 1 --2- 172.21.15.194:0/4255281220 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f90f0 unknown :-1 s=CLOSED pgs=299 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:41.596 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.604+0000 7f8ea9d07700 1 -- 172.21.15.194:0/4255281220 >> 172.21.15.194:0/4255281220 conn(0x7f8ea40f0b70 msgr2=0x7f8ea40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:41.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.604+0000 7f8ea9d07700 1 -- 172.21.15.194:0/4255281220 shutdown_connections 2024-01-27T23:30:41.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.604+0000 7f8ea9d07700 1 -- 172.21.15.194:0/4255281220 wait complete. 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.605+0000 7f8ea9d07700 1 Processor -- start 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.605+0000 7f8ea9d07700 1 -- start start 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.605+0000 7f8ea9d07700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f6b70 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.605+0000 7f8ea9d07700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8ea40f4ff0 con 0x7f8ea4060280 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.605+0000 7f8ea37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f6b70 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.605+0000 7f8ea37fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f6b70 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40416/0 (socket says 172.21.15.194:40416) 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.605+0000 7f8ea37fe700 1 -- 172.21.15.194:0/2054960455 learned_addr learned my addr 172.21.15.194:0/2054960455 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.606+0000 7f8ea37fe700 1 -- 172.21.15.194:0/2054960455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8e98005040 con 0x7f8ea4060280 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.606+0000 7f8ea37fe700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f6b70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f8e98009830 tx=0x7f8e9800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:41.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.606+0000 7f8e8bfff700 1 -- 172.21.15.194:0/2054960455 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e98009320 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.606+0000 7f8e8bfff700 1 -- 172.21.15.194:0/2054960455 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8e980136c0 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.606+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8ea40f5190 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.606+0000 7f8e8bfff700 1 -- 172.21.15.194:0/2054960455 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8e9801ee90 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.606+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8ea40f5620 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.607+0000 7f8e8bfff700 1 -- 172.21.15.194:0/2054960455 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f8e98014420 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.607+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8ea4059100 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.607+0000 7f8e8bfff700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8e8c02ff10 0x7f8e8c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.607+0000 7f8e8bfff700 1 -- 172.21.15.194:0/2054960455 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8e980412b0 con 0x7f8ea4060280 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.607+0000 7f8ea2ffd700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8e8c02ff10 0x7f8e8c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:41.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.607+0000 7f8ea2ffd700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8e8c02ff10 0x7f8e8c0323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8e940096a0 tx=0x7f8e94006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:41.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.611+0000 7f8e8bfff700 1 -- 172.21.15.194:0/2054960455 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8e9801c320 con 0x7f8ea4060280 2024-01-27T23:30:41.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.816+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8ea40fbf50 con 0x7f8ea4060280 2024-01-27T23:30:41.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.816+0000 7f8e8bfff700 1 -- 172.21.15.194:0/2054960455 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f8e9801c320 con 0x7f8ea4060280 2024-01-27T23:30:41.808 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:41.808 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8e8c02ff10 msgr2=0x7f8e8c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8e8c02ff10 0x7f8e8c0323d0 secure :-1 s=READY pgs=116 cs=0 l=1 rev1=1 rx=0x7f8e940096a0 tx=0x7f8e94006b40).stop 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 msgr2=0x7f8ea40f6b70 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f6b70 secure :-1 s=READY pgs=300 cs=0 l=1 rev1=1 rx=0x7f8e98009830 tx=0x7f8e9800b040).stop 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 shutdown_connections 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f8e8c02ff10 0x7f8e8c0323d0 unknown :-1 s=CLOSED pgs=116 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 --2- 172.21.15.194:0/2054960455 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8ea4060280 0x7f8ea40f6b70 unknown :-1 s=CLOSED pgs=300 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 >> 172.21.15.194:0/2054960455 conn(0x7f8ea40f0b70 msgr2=0x7f8ea40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 shutdown_connections 2024-01-27T23:30:41.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:41.818+0000 7f8ea9d07700 1 -- 172.21.15.194:0/2054960455 wait complete. 2024-01-27T23:30:41.811 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:42.449 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:41.307764+0000 mgr.smithi121.hcppvz (mgr.14162) 167 : cluster [DBG] pgmap v156: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:42.450 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:41.816563+0000 mon.smithi121 (mon.0) 241 : audit [DBG] from='client.? 172.21.15.194:0/2054960455' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:43.177 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:43.178 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:44.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.395+0000 7f5eac592700 1 -- 172.21.15.194:0/995475074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 msgr2=0x7f5ea40f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.395+0000 7f5eac592700 1 --2- 172.21.15.194:0/995475074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea40f9150 secure :-1 s=READY pgs=301 cs=0 l=1 rev1=1 rx=0x7f5e94009fb0 tx=0x7f5e94009b30).stop 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.395+0000 7f5eac592700 1 -- 172.21.15.194:0/995475074 shutdown_connections 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.395+0000 7f5eac592700 1 --2- 172.21.15.194:0/995475074 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea40f9150 unknown :-1 s=CLOSED pgs=301 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.395+0000 7f5eac592700 1 -- 172.21.15.194:0/995475074 >> 172.21.15.194:0/995475074 conn(0x7f5ea40f0b70 msgr2=0x7f5ea40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.395+0000 7f5eac592700 1 -- 172.21.15.194:0/995475074 shutdown_connections 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.395+0000 7f5eac592700 1 -- 172.21.15.194:0/995475074 wait complete. 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eac592700 1 Processor -- start 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eac592700 1 -- start start 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eac592700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea41066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:44.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eac592700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5ea4106bd0 con 0x7f5ea4060220 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eaa32e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea41066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eaa32e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea41066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40440/0 (socket says 172.21.15.194:40440) 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eaa32e700 1 -- 172.21.15.194:0/3158607695 learned_addr learned my addr 172.21.15.194:0/3158607695 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.396+0000 7f5eaa32e700 1 -- 172.21.15.194:0/3158607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5e94005040 con 0x7f5ea4060220 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.397+0000 7f5eaa32e700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea41066e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f5e94013510 tx=0x7f5e9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.397+0000 7f5e9affd700 1 -- 172.21.15.194:0/3158607695 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e94009320 con 0x7f5ea4060220 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.397+0000 7f5e9affd700 1 -- 172.21.15.194:0/3158607695 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5e94013f00 con 0x7f5ea4060220 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.397+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5ea4106dd0 con 0x7f5ea4060220 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.397+0000 7f5e9affd700 1 -- 172.21.15.194:0/3158607695 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5e94014dc0 con 0x7f5ea4060220 2024-01-27T23:30:44.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.397+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5ea4107260 con 0x7f5ea4060220 2024-01-27T23:30:44.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.397+0000 7f5e9affd700 1 -- 172.21.15.194:0/3158607695 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f5e9401d3d0 con 0x7f5ea4060220 2024-01-27T23:30:44.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.398+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5ea4045ad0 con 0x7f5ea4060220 2024-01-27T23:30:44.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.398+0000 7f5e9affd700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5e9002ff60 0x7f5e90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:44.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.398+0000 7f5e9affd700 1 -- 172.21.15.194:0/3158607695 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5e940410f0 con 0x7f5ea4060220 2024-01-27T23:30:44.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.398+0000 7f5ea9b2d700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5e9002ff60 0x7f5e90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:44.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.398+0000 7f5ea9b2d700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5e9002ff60 0x7f5e90032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f5ea00096a0 tx=0x7f5ea0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:44.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.402+0000 7f5e9affd700 1 -- 172.21.15.194:0/3158607695 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f5e940095f0 con 0x7f5ea4060220 2024-01-27T23:30:44.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:43.286817+0000 mon.smithi121 (mon.0) 242 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' 2024-01-27T23:30:44.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:30:43.288110+0000 mgr.smithi121.hcppvz (mgr.14162) 168 : cephadm [INF] Deploying daemon node-exporter.smithi121 on smithi121 2024-01-27T23:30:44.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:43.307992+0000 mgr.smithi121.hcppvz (mgr.14162) 169 : cluster [DBG] pgmap v157: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:44.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.605+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f5ea4108110 con 0x7f5ea4060220 2024-01-27T23:30:44.597 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.606+0000 7f5e9affd700 1 -- 172.21.15.194:0/3158607695 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5e9401cac0 con 0x7f5ea4060220 2024-01-27T23:30:44.598 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:44.598 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:44.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5e9002ff60 msgr2=0x7f5e90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:44.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5e9002ff60 0x7f5e90032420 secure :-1 s=READY pgs=117 cs=0 l=1 rev1=1 rx=0x7f5ea00096a0 tx=0x7f5ea0006b40).stop 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 msgr2=0x7f5ea41066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea41066e0 secure :-1 s=READY pgs=302 cs=0 l=1 rev1=1 rx=0x7f5e94013510 tx=0x7f5e9400b040).stop 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 shutdown_connections 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f5e9002ff60 0x7f5e90032420 unknown :-1 s=CLOSED pgs=117 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 --2- 172.21.15.194:0/3158607695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f5ea4060220 0x7f5ea41066e0 unknown :-1 s=CLOSED pgs=302 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.607+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 >> 172.21.15.194:0/3158607695 conn(0x7f5ea40f0b70 msgr2=0x7f5ea40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.608+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 shutdown_connections 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:44.608+0000 7f5eac592700 1 -- 172.21.15.194:0/3158607695 wait complete. 2024-01-27T23:30:44.600 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:45.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:45 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:44.606240+0000 mon.smithi121 (mon.0) 243 : audit [DBG] from='client.? 172.21.15.194:0/3158607695' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:45.984 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:45.984 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:46.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:45.308323+0000 mgr.smithi121.hcppvz (mgr.14162) 170 : cluster [DBG] pgmap v158: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:47.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 -- 172.21.15.194:0/1098562231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 msgr2=0x7fac740f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:47.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 --2- 172.21.15.194:0/1098562231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac740f6930 secure :-1 s=READY pgs=303 cs=0 l=1 rev1=1 rx=0x7fac64009fb0 tx=0x7fac64009b30).stop 2024-01-27T23:30:47.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 -- 172.21.15.194:0/1098562231 shutdown_connections 2024-01-27T23:30:47.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 --2- 172.21.15.194:0/1098562231 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac740f6930 unknown :-1 s=CLOSED pgs=303 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:47.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 -- 172.21.15.194:0/1098562231 >> 172.21.15.194:0/1098562231 conn(0x7fac740f1c70 msgr2=0x7fac740f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:47.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 -- 172.21.15.194:0/1098562231 shutdown_connections 2024-01-27T23:30:47.150 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 -- 172.21.15.194:0/1098562231 wait complete. 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.157+0000 7fac7a498700 1 Processor -- start 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac7a498700 1 -- start start 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac7a498700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac74106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac7a498700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fac74106c20 con 0x7fac740f6510 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac73fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac74106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac73fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac74106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:43046/0 (socket says 172.21.15.194:43046) 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac73fff700 1 -- 172.21.15.194:0/3710900953 learned_addr learned my addr 172.21.15.194:0/3710900953 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac73fff700 1 -- 172.21.15.194:0/3710900953 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fac64005040 con 0x7fac740f6510 2024-01-27T23:30:47.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.158+0000 7fac73fff700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac74106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fac64013510 tx=0x7fac6400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:47.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.159+0000 7fac70ff9700 1 -- 172.21.15.194:0/3710900953 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac64009320 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.159+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fac74106e20 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.159+0000 7fac70ff9700 1 -- 172.21.15.194:0/3710900953 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fac64013f00 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.159+0000 7fac70ff9700 1 -- 172.21.15.194:0/3710900953 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fac64014d90 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.159+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fac741072b0 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.160+0000 7fac70ff9700 1 -- 172.21.15.194:0/3710900953 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fac640095f0 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.160+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fac74045ad0 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.160+0000 7fac70ff9700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fac5c02ff60 0x7fac5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.160+0000 7fac70ff9700 1 -- 172.21.15.194:0/3710900953 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fac64040fe0 con 0x7fac740f6510 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.160+0000 7fac737fe700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fac5c02ff60 0x7fac5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:47.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.160+0000 7fac737fe700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fac5c02ff60 0x7fac5c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fac680096a0 tx=0x7fac68006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:47.156 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.164+0000 7fac70ff9700 1 -- 172.21.15.194:0/3710900953 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fac6401d7f0 con 0x7fac740f6510 2024-01-27T23:30:47.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.374+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fac740fbf60 con 0x7fac740f6510 2024-01-27T23:30:47.366 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.374+0000 7fac70ff9700 1 -- 172.21.15.194:0/3710900953 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fac64010230 con 0x7fac740f6510 2024-01-27T23:30:47.366 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:47.366 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:47.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fac5c02ff60 msgr2=0x7fac5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:47.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fac5c02ff60 0x7fac5c032420 secure :-1 s=READY pgs=118 cs=0 l=1 rev1=1 rx=0x7fac680096a0 tx=0x7fac68006b40).stop 2024-01-27T23:30:47.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 msgr2=0x7fac74106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:47.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac74106730 secure :-1 s=READY pgs=304 cs=0 l=1 rev1=1 rx=0x7fac64013510 tx=0x7fac6400b040).stop 2024-01-27T23:30:47.368 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 shutdown_connections 2024-01-27T23:30:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fac5c02ff60 0x7fac5c032420 unknown :-1 s=CLOSED pgs=118 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 --2- 172.21.15.194:0/3710900953 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fac740f6510 0x7fac74106730 unknown :-1 s=CLOSED pgs=304 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 >> 172.21.15.194:0/3710900953 conn(0x7fac740f1c70 msgr2=0x7fac740f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 shutdown_connections 2024-01-27T23:30:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:47.376+0000 7fac7a498700 1 -- 172.21.15.194:0/3710900953 wait complete. 2024-01-27T23:30:47.369 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:48.807 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:48.808 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:48.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:47.308665+0000 mgr.smithi121.hcppvz (mgr.14162) 171 : cluster [DBG] pgmap v159: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:48.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:47.374677+0000 mon.smithi121 (mon.0) 244 : audit [DBG] from='client.? 172.21.15.194:0/3710900953' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:49.985 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.991+0000 7f0a83ef0700 1 -- 172.21.15.194:0/1600220709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c0f5410 msgr2=0x7f0a7c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.991+0000 7f0a83ef0700 1 --2- 172.21.15.194:0/1600220709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c0f5410 0x7f0a7c0f5830 secure :-1 s=READY pgs=305 cs=0 l=1 rev1=1 rx=0x7f0a6c009fb0 tx=0x7f0a6c009b30).stop 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.991+0000 7f0a83ef0700 1 -- 172.21.15.194:0/1600220709 shutdown_connections 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.991+0000 7f0a83ef0700 1 --2- 172.21.15.194:0/1600220709 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c0f5410 0x7f0a7c0f5830 unknown :-1 s=CLOSED pgs=305 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.991+0000 7f0a83ef0700 1 -- 172.21.15.194:0/1600220709 >> 172.21.15.194:0/1600220709 conn(0x7f0a7c0f0b70 msgr2=0x7f0a7c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.991+0000 7f0a83ef0700 1 -- 172.21.15.194:0/1600220709 shutdown_connections 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.991+0000 7f0a83ef0700 1 -- 172.21.15.194:0/1600220709 wait complete. 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a83ef0700 1 Processor -- start 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a83ef0700 1 -- start start 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a83ef0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c106830 0x7f0a7c106c50 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:49.986 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a83ef0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0a7c107140 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a81c8c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c106830 0x7f0a7c106c50 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a81c8c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c106830 0x7f0a7c106c50 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:43060/0 (socket says 172.21.15.194:43060) 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a81c8c700 1 -- 172.21.15.194:0/3975113113 learned_addr learned my addr 172.21.15.194:0/3975113113 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a81c8c700 1 -- 172.21.15.194:0/3975113113 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0a6c005040 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.992+0000 7f0a81c8c700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c106830 0x7f0a7c106c50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f0a6c009f80 tx=0x7f0a6c009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.993+0000 7f0a727fc700 1 -- 172.21.15.194:0/3975113113 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a6c0136c0 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.993+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0a7c107340 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.993+0000 7f0a727fc700 1 -- 172.21.15.194:0/3975113113 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0a6c013e70 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.993+0000 7f0a727fc700 1 -- 172.21.15.194:0/3975113113 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0a6c014420 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.993+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0a7c107e90 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.993+0000 7f0a727fc700 1 -- 172.21.15.194:0/3975113113 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0a6c025060 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.993+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0a7c059100 con 0x7f0a7c106830 2024-01-27T23:30:49.987 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.994+0000 7f0a727fc700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0a6802ff60 0x7f0a68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:49.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.994+0000 7f0a727fc700 1 -- 172.21.15.194:0/3975113113 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0a6c040f00 con 0x7f0a7c106830 2024-01-27T23:30:49.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.994+0000 7f0a8148b700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0a6802ff60 0x7f0a68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:49.988 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.994+0000 7f0a8148b700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0a6802ff60 0x7f0a68032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0a780096a0 tx=0x7f0a78006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:49.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:49.998+0000 7f0a727fc700 1 -- 172.21.15.194:0/3975113113 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0a6c01cb30 con 0x7f0a7c106830 2024-01-27T23:30:50.198 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.206+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0a7c045ad0 con 0x7f0a7c106830 2024-01-27T23:30:50.199 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.207+0000 7f0a727fc700 1 -- 172.21.15.194:0/3975113113 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0a6c02d2d0 con 0x7f0a7c106830 2024-01-27T23:30:50.199 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:50.199 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:50.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0a6802ff60 msgr2=0x7f0a68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:50.201 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0a6802ff60 0x7f0a68032420 secure :-1 s=READY pgs=119 cs=0 l=1 rev1=1 rx=0x7f0a780096a0 tx=0x7f0a78006b40).stop 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c106830 msgr2=0x7f0a7c106c50 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c106830 0x7f0a7c106c50 secure :-1 s=READY pgs=306 cs=0 l=1 rev1=1 rx=0x7f0a6c009f80 tx=0x7f0a6c009b30).stop 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 shutdown_connections 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0a6802ff60 0x7f0a68032420 unknown :-1 s=CLOSED pgs=119 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 --2- 172.21.15.194:0/3975113113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0a7c106830 0x7f0a7c106c50 unknown :-1 s=CLOSED pgs=306 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 >> 172.21.15.194:0/3975113113 conn(0x7f0a7c0f0b70 msgr2=0x7f0a7c0f1700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 shutdown_connections 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:50.209+0000 7f0a83ef0700 1 -- 172.21.15.194:0/3975113113 wait complete. 2024-01-27T23:30:50.202 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:50.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:49.308981+0000 mgr.smithi121.hcppvz (mgr.14162) 172 : cluster [DBG] pgmap v160: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:50.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:50.207393+0000 mon.smithi121 (mon.0) 245 : audit [DBG] from='client.? 172.21.15.194:0/3975113113' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:51.587 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:51.587 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:52.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:51.309281+0000 mgr.smithi121.hcppvz (mgr.14162) 173 : cluster [DBG] pgmap v161: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.735+0000 7fc26d91d700 1 -- 172.21.15.194:0/98782984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 msgr2=0x7fc2680f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.735+0000 7fc26d91d700 1 --2- 172.21.15.194:0/98782984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc2680f66c0 secure :-1 s=READY pgs=307 cs=0 l=1 rev1=1 rx=0x7fc25c009fb0 tx=0x7fc25c009b30).stop 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 -- 172.21.15.194:0/98782984 shutdown_connections 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 --2- 172.21.15.194:0/98782984 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc2680f66c0 unknown :-1 s=CLOSED pgs=307 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 -- 172.21.15.194:0/98782984 >> 172.21.15.194:0/98782984 conn(0x7fc2680f1ac0 msgr2=0x7fc2680f3ee0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 -- 172.21.15.194:0/98782984 shutdown_connections 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 -- 172.21.15.194:0/98782984 wait complete. 2024-01-27T23:30:52.729 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 Processor -- start 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 -- start start 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc26806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.736+0000 7fc26d91d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc268069e20 con 0x7fc2680f62a0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc26c91b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc26806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc26c91b700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc26806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:43082/0 (socket says 172.21.15.194:43082) 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc26c91b700 1 -- 172.21.15.194:0/2021847634 learned_addr learned my addr 172.21.15.194:0/2021847634 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc26c91b700 1 -- 172.21.15.194:0/2021847634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc25c005040 con 0x7fc2680f62a0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc26c91b700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc26806b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc25c013510 tx=0x7fc25c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc2657fa700 1 -- 172.21.15.194:0/2021847634 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc25c009320 con 0x7fc2680f62a0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc2657fa700 1 -- 172.21.15.194:0/2021847634 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc25c013f00 con 0x7fc2680f62a0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc268069fc0 con 0x7fc2680f62a0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc2657fa700 1 -- 172.21.15.194:0/2021847634 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc25c014dc0 con 0x7fc2680f62a0 2024-01-27T23:30:52.730 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.737+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc26806a450 con 0x7fc2680f62a0 2024-01-27T23:30:52.731 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.738+0000 7fc2657fa700 1 -- 172.21.15.194:0/2021847634 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc25c0095f0 con 0x7fc2680f62a0 2024-01-27T23:30:52.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.738+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc268059100 con 0x7fc2680f62a0 2024-01-27T23:30:52.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.738+0000 7fc2657fa700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc25802ffb0 0x7fc258032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:52.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.738+0000 7fc2657fa700 1 -- 172.21.15.194:0/2021847634 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc25c041100 con 0x7fc2680f62a0 2024-01-27T23:30:52.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.738+0000 7fc267fff700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc25802ffb0 0x7fc258032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:52.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.739+0000 7fc267fff700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc25802ffb0 0x7fc258032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc2540096a0 tx=0x7fc254006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:52.735 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.743+0000 7fc2657fa700 1 -- 172.21.15.194:0/2021847634 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc25c014420 con 0x7fc2680f62a0 2024-01-27T23:30:52.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.951+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc2680fbce0 con 0x7fc2680f62a0 2024-01-27T23:30:52.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.951+0000 7fc2657fa700 1 -- 172.21.15.194:0/2021847634 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc25c010230 con 0x7fc2680f62a0 2024-01-27T23:30:52.944 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:52.944 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:52.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc25802ffb0 msgr2=0x7fc258032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:52.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc25802ffb0 0x7fc258032470 secure :-1 s=READY pgs=120 cs=0 l=1 rev1=1 rx=0x7fc2540096a0 tx=0x7fc254006b40).stop 2024-01-27T23:30:52.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 msgr2=0x7fc26806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:52.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc26806b9a0 secure :-1 s=READY pgs=308 cs=0 l=1 rev1=1 rx=0x7fc25c013510 tx=0x7fc25c00b040).stop 2024-01-27T23:30:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 shutdown_connections 2024-01-27T23:30:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc25802ffb0 0x7fc258032470 unknown :-1 s=CLOSED pgs=120 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 --2- 172.21.15.194:0/2021847634 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc2680f62a0 0x7fc26806b9a0 unknown :-1 s=CLOSED pgs=308 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 >> 172.21.15.194:0/2021847634 conn(0x7fc2680f1ac0 msgr2=0x7fc2680f2530 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 shutdown_connections 2024-01-27T23:30:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:52.953+0000 7fc26d91d700 1 -- 172.21.15.194:0/2021847634 wait complete. 2024-01-27T23:30:52.946 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:53.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:53 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:52.951967+0000 mon.smithi121 (mon.0) 246 : audit [DBG] from='client.? 172.21.15.194:0/2021847634' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:54.321 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:54.321 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:54.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:53.309546+0000 mgr.smithi121.hcppvz (mgr.14162) 174 : cluster [DBG] pgmap v162: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:55.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.474+0000 7efee8088700 1 -- 172.21.15.194:0/1664568090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 msgr2=0x7efee00f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.474+0000 7efee8088700 1 --2- 172.21.15.194:0/1664568090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee00f5830 secure :-1 s=READY pgs=309 cs=0 l=1 rev1=1 rx=0x7efed0009fb0 tx=0x7efed0009b30).stop 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.474+0000 7efee8088700 1 -- 172.21.15.194:0/1664568090 shutdown_connections 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.474+0000 7efee8088700 1 --2- 172.21.15.194:0/1664568090 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee00f5830 unknown :-1 s=CLOSED pgs=309 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.474+0000 7efee8088700 1 -- 172.21.15.194:0/1664568090 >> 172.21.15.194:0/1664568090 conn(0x7efee00f0b70 msgr2=0x7efee00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.474+0000 7efee8088700 1 -- 172.21.15.194:0/1664568090 shutdown_connections 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.474+0000 7efee8088700 1 -- 172.21.15.194:0/1664568090 wait complete. 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee8088700 1 Processor -- start 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee8088700 1 -- start start 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee8088700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee0105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee8088700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efee0105b20 con 0x7efee00f5410 2024-01-27T23:30:55.468 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee5e24700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee0105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee5e24700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee0105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32840/0 (socket says 172.21.15.194:32840) 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee5e24700 1 -- 172.21.15.194:0/342680768 learned_addr learned my addr 172.21.15.194:0/342680768 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee5e24700 1 -- 172.21.15.194:0/342680768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efed0005040 con 0x7efee00f5410 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.475+0000 7efee5e24700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee0105630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7efed0013510 tx=0x7efed000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.476+0000 7efed67fc700 1 -- 172.21.15.194:0/342680768 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efed0009320 con 0x7efee00f5410 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.476+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efee0105d20 con 0x7efee00f5410 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.476+0000 7efed67fc700 1 -- 172.21.15.194:0/342680768 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efed0013f00 con 0x7efee00f5410 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.476+0000 7efed67fc700 1 -- 172.21.15.194:0/342680768 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efed0014d90 con 0x7efee00f5410 2024-01-27T23:30:55.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.476+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efee01061b0 con 0x7efee00f5410 2024-01-27T23:30:55.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.476+0000 7efed67fc700 1 -- 172.21.15.194:0/342680768 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efed00095f0 con 0x7efee00f5410 2024-01-27T23:30:55.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.477+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efee0045ad0 con 0x7efee00f5410 2024-01-27T23:30:55.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.477+0000 7efed67fc700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efecc02ff60 0x7efecc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:55.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.477+0000 7efed67fc700 1 -- 172.21.15.194:0/342680768 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efed0042d90 con 0x7efee00f5410 2024-01-27T23:30:55.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.477+0000 7efee5623700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efecc02ff60 0x7efecc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:55.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.477+0000 7efee5623700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efecc02ff60 0x7efecc032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7efedc0096a0 tx=0x7efedc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:55.473 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.481+0000 7efed67fc700 1 -- 172.21.15.194:0/342680768 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efed001f7f0 con 0x7efee00f5410 2024-01-27T23:30:55.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.687+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efee00fae60 con 0x7efee00f5410 2024-01-27T23:30:55.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.688+0000 7efed67fc700 1 -- 172.21.15.194:0/342680768 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efed0011290 con 0x7efee00f5410 2024-01-27T23:30:55.680 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:55.680 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efecc02ff60 msgr2=0x7efecc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efecc02ff60 0x7efecc032420 secure :-1 s=READY pgs=121 cs=0 l=1 rev1=1 rx=0x7efedc0096a0 tx=0x7efedc006b40).stop 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 msgr2=0x7efee0105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee0105630 secure :-1 s=READY pgs=310 cs=0 l=1 rev1=1 rx=0x7efed0013510 tx=0x7efed000b040).stop 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 shutdown_connections 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efecc02ff60 0x7efecc032420 unknown :-1 s=CLOSED pgs=121 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 --2- 172.21.15.194:0/342680768 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efee00f5410 0x7efee0105630 unknown :-1 s=CLOSED pgs=310 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 >> 172.21.15.194:0/342680768 conn(0x7efee00f0b70 msgr2=0x7efee00f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:55.682 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 shutdown_connections 2024-01-27T23:30:55.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:55.690+0000 7efee8088700 1 -- 172.21.15.194:0/342680768 wait complete. 2024-01-27T23:30:55.683 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:56.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:55.309861+0000 mgr.smithi121.hcppvz (mgr.14162) 175 : cluster [DBG] pgmap v163: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:56.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:55.688459+0000 mon.smithi121 (mon.0) 247 : audit [DBG] from='client.? 172.21.15.194:0/342680768' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:30:57.061 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:57.061 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 -- 172.21.15.194:0/2204531658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 msgr2=0x7f7c9c0f8fa0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 --2- 172.21.15.194:0/2204531658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c0f8fa0 secure :-1 s=READY pgs=311 cs=0 l=1 rev1=1 rx=0x7f7c8c009fb0 tx=0x7f7c8c009b30).stop 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 -- 172.21.15.194:0/2204531658 shutdown_connections 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 --2- 172.21.15.194:0/2204531658 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c0f8fa0 unknown :-1 s=CLOSED pgs=311 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 -- 172.21.15.194:0/2204531658 >> 172.21.15.194:0/2204531658 conn(0x7f7c9c0f09a0 msgr2=0x7f7c9c0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 -- 172.21.15.194:0/2204531658 shutdown_connections 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 -- 172.21.15.194:0/2204531658 wait complete. 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 Processor -- start 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 -- start start 2024-01-27T23:30:58.166 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c1064f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:58.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.173+0000 7f7ca2247700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7c9c1069e0 con 0x7f7c9c060160 2024-01-27T23:30:58.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7ca1245700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c1064f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:58.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7ca1245700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c1064f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32860/0 (socket says 172.21.15.194:32860) 2024-01-27T23:30:58.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7ca1245700 1 -- 172.21.15.194:0/855044172 learned_addr learned my addr 172.21.15.194:0/855044172 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:30:58.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7ca1245700 1 -- 172.21.15.194:0/855044172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7c8c005040 con 0x7f7c9c060160 2024-01-27T23:30:58.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7ca1245700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c1064f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f7c8c013510 tx=0x7f7c8c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:58.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7c91ffb700 1 -- 172.21.15.194:0/855044172 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c8c009320 con 0x7f7c9c060160 2024-01-27T23:30:58.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7c9c106be0 con 0x7f7c9c060160 2024-01-27T23:30:58.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7c91ffb700 1 -- 172.21.15.194:0/855044172 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7c8c013f00 con 0x7f7c9c060160 2024-01-27T23:30:58.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.174+0000 7f7c91ffb700 1 -- 172.21.15.194:0/855044172 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7c8c014d90 con 0x7f7c9c060160 2024-01-27T23:30:58.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.175+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7c9c107070 con 0x7f7c9c060160 2024-01-27T23:30:58.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.175+0000 7f7c91ffb700 1 -- 172.21.15.194:0/855044172 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7c8c014420 con 0x7f7c9c060160 2024-01-27T23:30:58.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.175+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7c9c045ad0 con 0x7f7c9c060160 2024-01-27T23:30:58.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.175+0000 7f7c91ffb700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:30:58.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.175+0000 7f7c91ffb700 1 -- 172.21.15.194:0/855044172 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7c8c0410d0 con 0x7f7c9c060160 2024-01-27T23:30:58.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.176+0000 7f7ca0a44700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:30:58.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.176+0000 7f7ca0a44700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7c8802ff60 0x7f7c88032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7c980096a0 tx=0x7f7c98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:30:58.172 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.180+0000 7f7c91ffb700 1 -- 172.21.15.194:0/855044172 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7c8c0095f0 con 0x7f7c9c060160 2024-01-27T23:30:58.378 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.386+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7c9c107ec0 con 0x7f7c9c060160 2024-01-27T23:30:58.378 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.387+0000 7f7c91ffb700 1 -- 172.21.15.194:0/855044172 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7c8c01db70 con 0x7f7c9c060160 2024-01-27T23:30:58.379 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:30:58.379 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.388+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7c8802ff60 msgr2=0x7f7c88032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.388+0000 7f7ca2247700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7c8802ff60 0x7f7c88032420 secure :-1 s=READY pgs=122 cs=0 l=1 rev1=1 rx=0x7f7c980096a0 tx=0x7f7c98006b40).stop 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.388+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 msgr2=0x7f7c9c1064f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.388+0000 7f7ca2247700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c1064f0 secure :-1 s=READY pgs=312 cs=0 l=1 rev1=1 rx=0x7f7c8c013510 tx=0x7f7c8c00b040).stop 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.388+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 shutdown_connections 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.388+0000 7f7ca2247700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7c8802ff60 0x7f7c88032420 unknown :-1 s=CLOSED pgs=122 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.389+0000 7f7ca2247700 1 --2- 172.21.15.194:0/855044172 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7c9c060160 0x7f7c9c1064f0 unknown :-1 s=CLOSED pgs=312 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.389+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 >> 172.21.15.194:0/855044172 conn(0x7f7c9c0f09a0 msgr2=0x7f7c9c0f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.389+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 shutdown_connections 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:30:58.389+0000 7f7ca2247700 1 -- 172.21.15.194:0/855044172 wait complete. 2024-01-27T23:30:58.381 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:30:58.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:57.310253+0000 mgr.smithi121.hcppvz (mgr.14162) 176 : cluster [DBG] pgmap v164: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:30:59.767 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:30:59.768 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:30:59.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:30:59 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:30:58.387000+0000 mon.smithi121 (mon.0) 248 : audit [DBG] from='client.? 172.21.15.194:0/855044172' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:00.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:30:59.310510+0000 mgr.smithi121.hcppvz (mgr.14162) 177 : cluster [DBG] pgmap v165: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:00.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.867+0000 7fd7665f0700 1 -- 172.21.15.194:0/2861534788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 msgr2=0x7fd7600f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:00.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.867+0000 7fd7665f0700 1 --2- 172.21.15.194:0/2861534788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd7600f5830 secure :-1 s=READY pgs=313 cs=0 l=1 rev1=1 rx=0x7fd750009fb0 tx=0x7fd750009b30).stop 2024-01-27T23:31:00.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.867+0000 7fd7665f0700 1 -- 172.21.15.194:0/2861534788 shutdown_connections 2024-01-27T23:31:00.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.867+0000 7fd7665f0700 1 --2- 172.21.15.194:0/2861534788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd7600f5830 unknown :-1 s=CLOSED pgs=313 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:00.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.867+0000 7fd7665f0700 1 -- 172.21.15.194:0/2861534788 >> 172.21.15.194:0/2861534788 conn(0x7fd7600f0b70 msgr2=0x7fd7600f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:00.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.867+0000 7fd7665f0700 1 -- 172.21.15.194:0/2861534788 shutdown_connections 2024-01-27T23:31:00.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.867+0000 7fd7665f0700 1 -- 172.21.15.194:0/2861534788 wait complete. 2024-01-27T23:31:00.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.868+0000 7fd7665f0700 1 Processor -- start 2024-01-27T23:31:00.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.868+0000 7fd7665f0700 1 -- start start 2024-01-27T23:31:00.861 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.868+0000 7fd7665f0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd760109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.868+0000 7fd7665f0700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd760060220 con 0x7fd7600f5410 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.868+0000 7fd75ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd760109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.868+0000 7fd75ffff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd760109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32888/0 (socket says 172.21.15.194:32888) 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.868+0000 7fd75ffff700 1 -- 172.21.15.194:0/412789595 learned_addr learned my addr 172.21.15.194:0/412789595 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.869+0000 7fd75ffff700 1 -- 172.21.15.194:0/412789595 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd750005040 con 0x7fd7600f5410 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.869+0000 7fd75ffff700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd760109450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd750013510 tx=0x7fd75000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.869+0000 7fd75cff9700 1 -- 172.21.15.194:0/412789595 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd750009320 con 0x7fd7600f5410 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.869+0000 7fd75cff9700 1 -- 172.21.15.194:0/412789595 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd750013f00 con 0x7fd7600f5410 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.869+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd760060420 con 0x7fd7600f5410 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.869+0000 7fd75cff9700 1 -- 172.21.15.194:0/412789595 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd750014d90 con 0x7fd7600f5410 2024-01-27T23:31:00.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.869+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd760109b00 con 0x7fd7600f5410 2024-01-27T23:31:00.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.870+0000 7fd75cff9700 1 -- 172.21.15.194:0/412789595 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7500263a0 con 0x7fd7600f5410 2024-01-27T23:31:00.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.870+0000 7fd75cff9700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd74802ff60 0x7fd748032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:00.863 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.870+0000 7fd75cff9700 1 -- 172.21.15.194:0/412789595 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd7500410e0 con 0x7fd7600f5410 2024-01-27T23:31:00.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.870+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd760059100 con 0x7fd7600f5410 2024-01-27T23:31:00.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.870+0000 7fd75f7fe700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd74802ff60 0x7fd748032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:00.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.871+0000 7fd75f7fe700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd74802ff60 0x7fd748032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fd7540096a0 tx=0x7fd754006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:00.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:00.875+0000 7fd75cff9700 1 -- 172.21.15.194:0/412789595 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd750012920 con 0x7fd7600f5410 2024-01-27T23:31:01.080 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.088+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd76010a250 con 0x7fd7600f5410 2024-01-27T23:31:01.081 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.089+0000 7fd75cff9700 1 -- 172.21.15.194:0/412789595 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd75002b020 con 0x7fd7600f5410 2024-01-27T23:31:01.081 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:01.081 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:01.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd74802ff60 msgr2=0x7fd748032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd74802ff60 0x7fd748032420 secure :-1 s=READY pgs=123 cs=0 l=1 rev1=1 rx=0x7fd7540096a0 tx=0x7fd754006b40).stop 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 msgr2=0x7fd760109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd760109450 secure :-1 s=READY pgs=314 cs=0 l=1 rev1=1 rx=0x7fd750013510 tx=0x7fd75000b040).stop 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 shutdown_connections 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd74802ff60 0x7fd748032420 unknown :-1 s=CLOSED pgs=123 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 --2- 172.21.15.194:0/412789595 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd7600f5410 0x7fd760109450 unknown :-1 s=CLOSED pgs=314 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 >> 172.21.15.194:0/412789595 conn(0x7fd7600f0b70 msgr2=0x7fd7600f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.090+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 shutdown_connections 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:01.091+0000 7fd7665f0700 1 -- 172.21.15.194:0/412789595 wait complete. 2024-01-27T23:31:01.083 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:01.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:01 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:01.089085+0000 mon.smithi121 (mon.0) 249 : audit [DBG] from='client.? 172.21.15.194:0/412789595' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:02.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:02.452 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:02.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:01.310835+0000 mgr.smithi121.hcppvz (mgr.14162) 178 : cluster [DBG] pgmap v166: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:03.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.548+0000 7f2b77c25700 1 -- 172.21.15.194:0/432345343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 msgr2=0x7f2b700f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:03.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.548+0000 7f2b77c25700 1 --2- 172.21.15.194:0/432345343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b700f5830 secure :-1 s=READY pgs=315 cs=0 l=1 rev1=1 rx=0x7f2b60009fb0 tx=0x7f2b60009b30).stop 2024-01-27T23:31:03.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.548+0000 7f2b77c25700 1 -- 172.21.15.194:0/432345343 shutdown_connections 2024-01-27T23:31:03.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.548+0000 7f2b77c25700 1 --2- 172.21.15.194:0/432345343 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b700f5830 unknown :-1 s=CLOSED pgs=315 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:03.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.548+0000 7f2b77c25700 1 -- 172.21.15.194:0/432345343 >> 172.21.15.194:0/432345343 conn(0x7f2b700f0b70 msgr2=0x7f2b700f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.548+0000 7f2b77c25700 1 -- 172.21.15.194:0/432345343 shutdown_connections 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.548+0000 7f2b77c25700 1 -- 172.21.15.194:0/432345343 wait complete. 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b77c25700 1 Processor -- start 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b77c25700 1 -- start start 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b77c25700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b70106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b77c25700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f2b70106c10 con 0x7f2b700f5410 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b759c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b70106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b759c1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b70106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:32908/0 (socket says 172.21.15.194:32908) 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b759c1700 1 -- 172.21.15.194:0/2302430289 learned_addr learned my addr 172.21.15.194:0/2302430289 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.549+0000 7f2b759c1700 1 -- 172.21.15.194:0/2302430289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f2b60005040 con 0x7f2b700f5410 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.550+0000 7f2b759c1700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b70106720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2b60014000 tx=0x7f2b6000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.550+0000 7f2b667fc700 1 -- 172.21.15.194:0/2302430289 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b600095f0 con 0x7f2b700f5410 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.550+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f2b70106e10 con 0x7f2b700f5410 2024-01-27T23:31:03.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.550+0000 7f2b667fc700 1 -- 172.21.15.194:0/2302430289 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f2b60007e70 con 0x7f2b700f5410 2024-01-27T23:31:03.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.550+0000 7f2b667fc700 1 -- 172.21.15.194:0/2302430289 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f2b60014ea0 con 0x7f2b700f5410 2024-01-27T23:31:03.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.550+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f2b701072a0 con 0x7f2b700f5410 2024-01-27T23:31:03.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.551+0000 7f2b667fc700 1 -- 172.21.15.194:0/2302430289 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f2b60025070 con 0x7f2b700f5410 2024-01-27T23:31:03.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.551+0000 7f2b667fc700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2b5c02ff60 0x7f2b5c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:03.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.551+0000 7f2b667fc700 1 -- 172.21.15.194:0/2302430289 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f2b600409f0 con 0x7f2b700f5410 2024-01-27T23:31:03.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.551+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f2b70045ad0 con 0x7f2b700f5410 2024-01-27T23:31:03.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.551+0000 7f2b751c0700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2b5c02ff60 0x7f2b5c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:03.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.551+0000 7f2b751c0700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2b5c02ff60 0x7f2b5c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2b6c0096a0 tx=0x7f2b6c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:03.547 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.556+0000 7f2b667fc700 1 -- 172.21.15.194:0/2302430289 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f2b600079c0 con 0x7f2b700f5410 2024-01-27T23:31:03.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.763+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f2b701080d0 con 0x7f2b700f5410 2024-01-27T23:31:03.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.764+0000 7f2b667fc700 1 -- 172.21.15.194:0/2302430289 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f2b6002d090 con 0x7f2b700f5410 2024-01-27T23:31:03.756 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:03.756 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.765+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2b5c02ff60 msgr2=0x7f2b5c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.765+0000 7f2b77c25700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2b5c02ff60 0x7f2b5c032420 secure :-1 s=READY pgs=124 cs=0 l=1 rev1=1 rx=0x7f2b6c0096a0 tx=0x7f2b6c006b40).stop 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.765+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 msgr2=0x7f2b70106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.765+0000 7f2b77c25700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b70106720 secure :-1 s=READY pgs=316 cs=0 l=1 rev1=1 rx=0x7f2b60014000 tx=0x7f2b6000b040).stop 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.766+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 shutdown_connections 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.766+0000 7f2b77c25700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f2b5c02ff60 0x7f2b5c032420 unknown :-1 s=CLOSED pgs=124 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.766+0000 7f2b77c25700 1 --2- 172.21.15.194:0/2302430289 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f2b700f5410 0x7f2b70106720 unknown :-1 s=CLOSED pgs=316 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.766+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 >> 172.21.15.194:0/2302430289 conn(0x7f2b700f0b70 msgr2=0x7f2b7005fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.766+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 shutdown_connections 2024-01-27T23:31:03.758 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:03.766+0000 7f2b77c25700 1 -- 172.21.15.194:0/2302430289 wait complete. 2024-01-27T23:31:03.759 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:04.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:03.311160+0000 mgr.smithi121.hcppvz (mgr.14162) 179 : cluster [DBG] pgmap v167: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:04.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:03.764245+0000 mon.smithi121 (mon.0) 250 : audit [DBG] from='client.? 172.21.15.194:0/2302430289' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:05.142 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:05.142 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:06.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 -- 172.21.15.194:0/2556198717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 msgr2=0x7efd940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:06.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 --2- 172.21.15.194:0/2556198717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd940f6930 secure :-1 s=READY pgs=317 cs=0 l=1 rev1=1 rx=0x7efd84009fb0 tx=0x7efd84009b30).stop 2024-01-27T23:31:06.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 -- 172.21.15.194:0/2556198717 shutdown_connections 2024-01-27T23:31:06.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 --2- 172.21.15.194:0/2556198717 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd940f6930 unknown :-1 s=CLOSED pgs=317 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:06.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 -- 172.21.15.194:0/2556198717 >> 172.21.15.194:0/2556198717 conn(0x7efd940f1c70 msgr2=0x7efd940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:06.293 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 -- 172.21.15.194:0/2556198717 shutdown_connections 2024-01-27T23:31:06.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 -- 172.21.15.194:0/2556198717 wait complete. 2024-01-27T23:31:06.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 Processor -- start 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.301+0000 7efd99e9a700 1 -- start start 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.302+0000 7efd99e9a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd94106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.302+0000 7efd99e9a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7efd94106c20 con 0x7efd940f6510 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.302+0000 7efd937fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd94106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.302+0000 7efd937fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd94106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52398/0 (socket says 172.21.15.194:52398) 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.302+0000 7efd937fe700 1 -- 172.21.15.194:0/3906778988 learned_addr learned my addr 172.21.15.194:0/3906778988 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.302+0000 7efd937fe700 1 -- 172.21.15.194:0/3906778988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7efd84005040 con 0x7efd940f6510 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.302+0000 7efd937fe700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd94106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7efd84013510 tx=0x7efd8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.303+0000 7efd7bfff700 1 -- 172.21.15.194:0/3906778988 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd84009320 con 0x7efd940f6510 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.303+0000 7efd7bfff700 1 -- 172.21.15.194:0/3906778988 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7efd84013f00 con 0x7efd940f6510 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.303+0000 7efd7bfff700 1 -- 172.21.15.194:0/3906778988 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7efd84014d90 con 0x7efd940f6510 2024-01-27T23:31:06.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.303+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7efd94106e20 con 0x7efd940f6510 2024-01-27T23:31:06.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.303+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7efd941072b0 con 0x7efd940f6510 2024-01-27T23:31:06.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.304+0000 7efd7bfff700 1 -- 172.21.15.194:0/3906778988 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7efd840095f0 con 0x7efd940f6510 2024-01-27T23:31:06.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.304+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7efd94045ad0 con 0x7efd940f6510 2024-01-27T23:31:06.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.304+0000 7efd7bfff700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efd7c030000 0x7efd7c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:06.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.304+0000 7efd7bfff700 1 -- 172.21.15.194:0/3906778988 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7efd84041070 con 0x7efd940f6510 2024-01-27T23:31:06.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.304+0000 7efd92ffd700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efd7c030000 0x7efd7c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:06.298 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.305+0000 7efd92ffd700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efd7c030000 0x7efd7c0324c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7efd880096a0 tx=0x7efd88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:06.301 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.309+0000 7efd7bfff700 1 -- 172.21.15.194:0/3906778988 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7efd8406d2b0 con 0x7efd940f6510 2024-01-27T23:31:06.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.514+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7efd940fbf60 con 0x7efd940f6510 2024-01-27T23:31:06.507 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.515+0000 7efd7bfff700 1 -- 172.21.15.194:0/3906778988 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7efd84014420 con 0x7efd940f6510 2024-01-27T23:31:06.507 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:06.507 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efd7c030000 msgr2=0x7efd7c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efd7c030000 0x7efd7c0324c0 secure :-1 s=READY pgs=125 cs=0 l=1 rev1=1 rx=0x7efd880096a0 tx=0x7efd88006b40).stop 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 msgr2=0x7efd94106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd94106730 secure :-1 s=READY pgs=318 cs=0 l=1 rev1=1 rx=0x7efd84013510 tx=0x7efd8400b040).stop 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 shutdown_connections 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7efd7c030000 0x7efd7c0324c0 unknown :-1 s=CLOSED pgs=125 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 --2- 172.21.15.194:0/3906778988 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7efd940f6510 0x7efd94106730 unknown :-1 s=CLOSED pgs=318 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 >> 172.21.15.194:0/3906778988 conn(0x7efd940f1c70 msgr2=0x7efd940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 shutdown_connections 2024-01-27T23:31:06.509 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:06.516+0000 7efd99e9a700 1 -- 172.21.15.194:0/3906778988 wait complete. 2024-01-27T23:31:06.510 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:06.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:05.311495+0000 mgr.smithi121.hcppvz (mgr.14162) 180 : cluster [DBG] pgmap v168: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:07.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:07 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:06.514999+0000 mon.smithi121 (mon.0) 251 : audit [DBG] from='client.? 172.21.15.194:0/3906778988' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:07.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:07.923 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:08.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:07.311774+0000 mgr.smithi121.hcppvz (mgr.14162) 181 : cluster [DBG] pgmap v169: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:09.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.045+0000 7f3c4cf61700 1 -- 172.21.15.194:0/4169631867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 msgr2=0x7f3c480f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:09.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.045+0000 7f3c4cf61700 1 --2- 172.21.15.194:0/4169631867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c480f6930 secure :-1 s=READY pgs=319 cs=0 l=1 rev1=1 rx=0x7f3c3c009fb0 tx=0x7f3c3c009b30).stop 2024-01-27T23:31:09.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 -- 172.21.15.194:0/4169631867 shutdown_connections 2024-01-27T23:31:09.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 --2- 172.21.15.194:0/4169631867 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c480f6930 unknown :-1 s=CLOSED pgs=319 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:09.039 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 -- 172.21.15.194:0/4169631867 >> 172.21.15.194:0/4169631867 conn(0x7f3c480f1c70 msgr2=0x7f3c480f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 -- 172.21.15.194:0/4169631867 shutdown_connections 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 -- 172.21.15.194:0/4169631867 wait complete. 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 Processor -- start 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 -- start start 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c48106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4cf61700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3c48106c20 con 0x7f3c480f6510 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c48106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c48106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52412/0 (socket says 172.21.15.194:52412) 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.046+0000 7f3c4659c700 1 -- 172.21.15.194:0/2597441139 learned_addr learned my addr 172.21.15.194:0/2597441139 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.047+0000 7f3c4659c700 1 -- 172.21.15.194:0/2597441139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3c3c005040 con 0x7f3c480f6510 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.047+0000 7f3c4659c700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c48106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f3c3c013510 tx=0x7f3c3c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.047+0000 7f3c36ffd700 1 -- 172.21.15.194:0/2597441139 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c3c009320 con 0x7f3c480f6510 2024-01-27T23:31:09.040 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.047+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3c48106e20 con 0x7f3c480f6510 2024-01-27T23:31:09.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.047+0000 7f3c36ffd700 1 -- 172.21.15.194:0/2597441139 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3c3c013f00 con 0x7f3c480f6510 2024-01-27T23:31:09.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.047+0000 7f3c36ffd700 1 -- 172.21.15.194:0/2597441139 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3c3c014d90 con 0x7f3c480f6510 2024-01-27T23:31:09.041 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.047+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3c481072b0 con 0x7f3c480f6510 2024-01-27T23:31:09.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.048+0000 7f3c36ffd700 1 -- 172.21.15.194:0/2597441139 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3c3c014420 con 0x7f3c480f6510 2024-01-27T23:31:09.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.048+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3c48045ad0 con 0x7f3c480f6510 2024-01-27T23:31:09.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.048+0000 7f3c36ffd700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3c3002ff60 0x7f3c30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:09.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.048+0000 7f3c36ffd700 1 -- 172.21.15.194:0/2597441139 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3c3c041000 con 0x7f3c480f6510 2024-01-27T23:31:09.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.048+0000 7f3c45d9b700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3c3002ff60 0x7f3c30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:09.042 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.049+0000 7f3c45d9b700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3c3002ff60 0x7f3c30032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f3c380096a0 tx=0x7f3c38006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:09.044 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.052+0000 7f3c36ffd700 1 -- 172.21.15.194:0/2597441139 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3c3c0095f0 con 0x7f3c480f6510 2024-01-27T23:31:09.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.258+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3c480fbf60 con 0x7f3c480f6510 2024-01-27T23:31:09.250 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.258+0000 7f3c36ffd700 1 -- 172.21.15.194:0/2597441139 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f3c3c01db70 con 0x7f3c480f6510 2024-01-27T23:31:09.251 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:09.251 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3c3002ff60 msgr2=0x7f3c30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3c3002ff60 0x7f3c30032420 secure :-1 s=READY pgs=126 cs=0 l=1 rev1=1 rx=0x7f3c380096a0 tx=0x7f3c38006b40).stop 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 msgr2=0x7f3c48106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c48106730 secure :-1 s=READY pgs=320 cs=0 l=1 rev1=1 rx=0x7f3c3c013510 tx=0x7f3c3c00b040).stop 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 shutdown_connections 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f3c3002ff60 0x7f3c30032420 unknown :-1 s=CLOSED pgs=126 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 --2- 172.21.15.194:0/2597441139 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3c480f6510 0x7f3c48106730 unknown :-1 s=CLOSED pgs=320 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 >> 172.21.15.194:0/2597441139 conn(0x7f3c480f1c70 msgr2=0x7f3c480f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 shutdown_connections 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:09.260+0000 7f3c4cf61700 1 -- 172.21.15.194:0/2597441139 wait complete. 2024-01-27T23:31:09.253 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:09.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:09.258670+0000 mon.smithi121 (mon.0) 252 : audit [DBG] from='client.? 172.21.15.194:0/2597441139' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:10.622 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:10.622 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:10.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:09.312025+0000 mgr.smithi121.hcppvz (mgr.14162) 182 : cluster [DBG] pgmap v170: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:11.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 -- 172.21.15.194:0/3978054068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 msgr2=0x7fcf540f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 --2- 172.21.15.194:0/3978054068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf540f66c0 secure :-1 s=READY pgs=321 cs=0 l=1 rev1=1 rx=0x7fcf48009fb0 tx=0x7fcf48009b30).stop 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 -- 172.21.15.194:0/3978054068 shutdown_connections 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 --2- 172.21.15.194:0/3978054068 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf540f66c0 unknown :-1 s=CLOSED pgs=321 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 -- 172.21.15.194:0/3978054068 >> 172.21.15.194:0/3978054068 conn(0x7fcf540f1aa0 msgr2=0x7fcf540f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 -- 172.21.15.194:0/3978054068 shutdown_connections 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 -- 172.21.15.194:0/3978054068 wait complete. 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.811+0000 7fcf5359e700 1 Processor -- start 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5359e700 1 -- start start 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf541064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5359e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcf541069a0 con 0x7fcf540f62a0 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf541064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf541064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52426/0 (socket says 172.21.15.194:52426) 2024-01-27T23:31:11.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5259c700 1 -- 172.21.15.194:0/3684137142 learned_addr learned my addr 172.21.15.194:0/3684137142 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5259c700 1 -- 172.21.15.194:0/3684137142 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcf48005040 con 0x7fcf540f62a0 2024-01-27T23:31:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.812+0000 7fcf5259c700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf541064b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fcf48009830 tx=0x7fcf4800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.813+0000 7fcf42ffd700 1 -- 172.21.15.194:0/3684137142 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf48009320 con 0x7fcf540f62a0 2024-01-27T23:31:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.813+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcf54106ba0 con 0x7fcf540f62a0 2024-01-27T23:31:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.813+0000 7fcf42ffd700 1 -- 172.21.15.194:0/3684137142 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcf480136c0 con 0x7fcf540f62a0 2024-01-27T23:31:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.813+0000 7fcf42ffd700 1 -- 172.21.15.194:0/3684137142 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcf4801ee90 con 0x7fcf540f62a0 2024-01-27T23:31:11.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.813+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcf54107030 con 0x7fcf540f62a0 2024-01-27T23:31:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.813+0000 7fcf42ffd700 1 -- 172.21.15.194:0/3684137142 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcf4801e420 con 0x7fcf540f62a0 2024-01-27T23:31:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.813+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcf54045ad0 con 0x7fcf540f62a0 2024-01-27T23:31:11.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.814+0000 7fcf42ffd700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcf3c02ff10 0x7fcf3c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:11.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.814+0000 7fcf42ffd700 1 -- 172.21.15.194:0/3684137142 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcf48040f80 con 0x7fcf540f62a0 2024-01-27T23:31:11.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.814+0000 7fcf51d9b700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcf3c02ff10 0x7fcf3c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:11.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.814+0000 7fcf51d9b700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcf3c02ff10 0x7fcf3c0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcf440096a0 tx=0x7fcf44006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:11.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:11.818+0000 7fcf42ffd700 1 -- 172.21.15.194:0/3684137142 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcf4806d2b0 con 0x7fcf540f62a0 2024-01-27T23:31:12.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.024+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcf54107ee0 con 0x7fcf540f62a0 2024-01-27T23:31:12.018 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.025+0000 7fcf42ffd700 1 -- 172.21.15.194:0/3684137142 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcf480095f0 con 0x7fcf540f62a0 2024-01-27T23:31:12.019 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:12.019 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:12.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcf3c02ff10 msgr2=0x7fcf3c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:12.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcf3c02ff10 0x7fcf3c0323d0 secure :-1 s=READY pgs=127 cs=0 l=1 rev1=1 rx=0x7fcf440096a0 tx=0x7fcf44006b40).stop 2024-01-27T23:31:12.020 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 msgr2=0x7fcf541064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf541064b0 secure :-1 s=READY pgs=322 cs=0 l=1 rev1=1 rx=0x7fcf48009830 tx=0x7fcf4800b040).stop 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 shutdown_connections 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcf3c02ff10 0x7fcf3c0323d0 unknown :-1 s=CLOSED pgs=127 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 --2- 172.21.15.194:0/3684137142 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcf540f62a0 0x7fcf541064b0 unknown :-1 s=CLOSED pgs=322 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 >> 172.21.15.194:0/3684137142 conn(0x7fcf540f1aa0 msgr2=0x7fcf540f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 shutdown_connections 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:12.026+0000 7fcf5359e700 1 -- 172.21.15.194:0/3684137142 wait complete. 2024-01-27T23:31:12.021 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:12.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:11.312367+0000 mgr.smithi121.hcppvz (mgr.14162) 183 : cluster [DBG] pgmap v171: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:12.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:12.026196+0000 mon.smithi121 (mon.0) 253 : audit [DBG] from='client.? 172.21.15.194:0/3684137142' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:13.397 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:13.398 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:14.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.501+0000 7fa14f377700 1 -- 172.21.15.194:0/1657339872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 msgr2=0x7fa1480f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:14.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.501+0000 7fa14f377700 1 --2- 172.21.15.194:0/1657339872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa1480f5830 secure :-1 s=READY pgs=323 cs=0 l=1 rev1=1 rx=0x7fa138009fb0 tx=0x7fa138009b30).stop 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.501+0000 7fa14f377700 1 -- 172.21.15.194:0/1657339872 shutdown_connections 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.501+0000 7fa14f377700 1 --2- 172.21.15.194:0/1657339872 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa1480f5830 unknown :-1 s=CLOSED pgs=323 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.501+0000 7fa14f377700 1 -- 172.21.15.194:0/1657339872 >> 172.21.15.194:0/1657339872 conn(0x7fa1480f0b70 msgr2=0x7fa1480f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.502+0000 7fa14f377700 1 -- 172.21.15.194:0/1657339872 shutdown_connections 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.502+0000 7fa14f377700 1 -- 172.21.15.194:0/1657339872 wait complete. 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.502+0000 7fa14f377700 1 Processor -- start 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.502+0000 7fa14f377700 1 -- start start 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.502+0000 7fa14f377700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa148105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.502+0000 7fa14f377700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa148105b20 con 0x7fa1480f5410 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.503+0000 7fa14d113700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa148105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:14.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.503+0000 7fa14d113700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa148105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52446/0 (socket says 172.21.15.194:52446) 2024-01-27T23:31:14.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.503+0000 7fa14d113700 1 -- 172.21.15.194:0/3559618621 learned_addr learned my addr 172.21.15.194:0/3559618621 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:14.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.503+0000 7fa14d113700 1 -- 172.21.15.194:0/3559618621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa138005040 con 0x7fa1480f5410 2024-01-27T23:31:14.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.503+0000 7fa14d113700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa148105630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fa138013510 tx=0x7fa13800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:14.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.503+0000 7fa13dffb700 1 -- 172.21.15.194:0/3559618621 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa138009320 con 0x7fa1480f5410 2024-01-27T23:31:14.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.503+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa148105d20 con 0x7fa1480f5410 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.504+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa1481061b0 con 0x7fa1480f5410 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.504+0000 7fa13dffb700 1 -- 172.21.15.194:0/3559618621 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa138013f00 con 0x7fa1480f5410 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.504+0000 7fa13dffb700 1 -- 172.21.15.194:0/3559618621 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa138014d60 con 0x7fa1480f5410 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.504+0000 7fa13dffb700 1 -- 172.21.15.194:0/3559618621 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa1380095f0 con 0x7fa1480f5410 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.504+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa148045ad0 con 0x7fa1480f5410 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.504+0000 7fa13dffb700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa13402ff60 0x7fa134032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.505+0000 7fa13dffb700 1 -- 172.21.15.194:0/3559618621 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa1380410d0 con 0x7fa1480f5410 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.505+0000 7fa14c912700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa13402ff60 0x7fa134032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:14.499 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.505+0000 7fa14c912700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa13402ff60 0x7fa134032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa1440096a0 tx=0x7fa144006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:14.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.509+0000 7fa13dffb700 1 -- 172.21.15.194:0/3559618621 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa138014420 con 0x7fa1480f5410 2024-01-27T23:31:14.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.715+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa1480fae60 con 0x7fa1480f5410 2024-01-27T23:31:14.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.716+0000 7fa13dffb700 1 -- 172.21.15.194:0/3559618621 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa138012980 con 0x7fa1480f5410 2024-01-27T23:31:14.709 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:14.709 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa13402ff60 msgr2=0x7fa134032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa13402ff60 0x7fa134032420 secure :-1 s=READY pgs=128 cs=0 l=1 rev1=1 rx=0x7fa1440096a0 tx=0x7fa144006b40).stop 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 msgr2=0x7fa148105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa148105630 secure :-1 s=READY pgs=324 cs=0 l=1 rev1=1 rx=0x7fa138013510 tx=0x7fa13800b040).stop 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 shutdown_connections 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa13402ff60 0x7fa134032420 unknown :-1 s=CLOSED pgs=128 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 --2- 172.21.15.194:0/3559618621 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa1480f5410 0x7fa148105630 unknown :-1 s=CLOSED pgs=324 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 >> 172.21.15.194:0/3559618621 conn(0x7fa1480f0b70 msgr2=0x7fa1480f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 shutdown_connections 2024-01-27T23:31:14.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:14.717+0000 7fa14f377700 1 -- 172.21.15.194:0/3559618621 wait complete. 2024-01-27T23:31:14.712 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:14.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:13.312657+0000 mgr.smithi121.hcppvz (mgr.14162) 184 : cluster [DBG] pgmap v172: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:15.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:15 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:14.717063+0000 mon.smithi121 (mon.0) 254 : audit [DBG] from='client.? 172.21.15.194:0/3559618621' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:16.085 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:16.085 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:16.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:15.312917+0000 mgr.smithi121.hcppvz (mgr.14162) 185 : cluster [DBG] pgmap v173: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:17.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.252+0000 7f9a999f3700 1 -- 172.21.15.194:0/2457929818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 msgr2=0x7f9a940f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:17.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.252+0000 7f9a999f3700 1 --2- 172.21.15.194:0/2457929818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a940f9150 secure :-1 s=READY pgs=325 cs=0 l=1 rev1=1 rx=0x7f9a84009fb0 tx=0x7f9a84009b30).stop 2024-01-27T23:31:17.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.252+0000 7f9a999f3700 1 -- 172.21.15.194:0/2457929818 shutdown_connections 2024-01-27T23:31:17.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.252+0000 7f9a999f3700 1 --2- 172.21.15.194:0/2457929818 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a940f9150 unknown :-1 s=CLOSED pgs=325 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:17.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.252+0000 7f9a999f3700 1 -- 172.21.15.194:0/2457929818 >> 172.21.15.194:0/2457929818 conn(0x7f9a940f0b70 msgr2=0x7f9a940f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:17.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.252+0000 7f9a999f3700 1 -- 172.21.15.194:0/2457929818 shutdown_connections 2024-01-27T23:31:17.246 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.252+0000 7f9a999f3700 1 -- 172.21.15.194:0/2457929818 wait complete. 2024-01-27T23:31:17.247 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.253+0000 7f9a999f3700 1 Processor -- start 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.253+0000 7f9a999f3700 1 -- start start 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.253+0000 7f9a999f3700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a941066a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.253+0000 7f9a999f3700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9a94106b90 con 0x7f9a94060220 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.253+0000 7f9a92ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a941066a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.253+0000 7f9a92ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a941066a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:54622/0 (socket says 172.21.15.194:54622) 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.253+0000 7f9a92ffd700 1 -- 172.21.15.194:0/1184248628 learned_addr learned my addr 172.21.15.194:0/1184248628 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.254+0000 7f9a92ffd700 1 -- 172.21.15.194:0/1184248628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9a84005040 con 0x7f9a94060220 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.254+0000 7f9a92ffd700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a941066a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f9a84009830 tx=0x7f9a8400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.254+0000 7f9a7b7fe700 1 -- 172.21.15.194:0/1184248628 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a84009320 con 0x7f9a94060220 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.254+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9a94106d90 con 0x7f9a94060220 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.254+0000 7f9a7b7fe700 1 -- 172.21.15.194:0/1184248628 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9a840136c0 con 0x7f9a94060220 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.254+0000 7f9a7b7fe700 1 -- 172.21.15.194:0/1184248628 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9a8401ee90 con 0x7f9a94060220 2024-01-27T23:31:17.248 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.254+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9a94107220 con 0x7f9a94060220 2024-01-27T23:31:17.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.255+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9a94045ad0 con 0x7f9a94060220 2024-01-27T23:31:17.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.255+0000 7f9a7b7fe700 1 -- 172.21.15.194:0/1184248628 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9a8401e420 con 0x7f9a94060220 2024-01-27T23:31:17.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.255+0000 7f9a7b7fe700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9a7c02ff10 0x7f9a7c0323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:17.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.255+0000 7f9a7b7fe700 1 -- 172.21.15.194:0/1184248628 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9a840410a0 con 0x7f9a94060220 2024-01-27T23:31:17.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.255+0000 7f9a927fc700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9a7c02ff10 0x7f9a7c0323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:17.249 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.256+0000 7f9a927fc700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9a7c02ff10 0x7f9a7c0323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9a880096a0 tx=0x7f9a88006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:17.253 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.260+0000 7f9a7b7fe700 1 -- 172.21.15.194:0/1184248628 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9a8401c790 con 0x7f9a94060220 2024-01-27T23:31:17.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.470+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9a94108070 con 0x7f9a94060220 2024-01-27T23:31:17.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.471+0000 7f9a7b7fe700 1 -- 172.21.15.194:0/1184248628 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9a8402c2d0 con 0x7f9a94060220 2024-01-27T23:31:17.464 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:17.464 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9a7c02ff10 msgr2=0x7f9a7c0323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9a7c02ff10 0x7f9a7c0323d0 secure :-1 s=READY pgs=129 cs=0 l=1 rev1=1 rx=0x7f9a880096a0 tx=0x7f9a88006b40).stop 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 msgr2=0x7f9a941066a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a941066a0 secure :-1 s=READY pgs=326 cs=0 l=1 rev1=1 rx=0x7f9a84009830 tx=0x7f9a8400b040).stop 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 shutdown_connections 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9a7c02ff10 0x7f9a7c0323d0 unknown :-1 s=CLOSED pgs=129 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 --2- 172.21.15.194:0/1184248628 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9a94060220 0x7f9a941066a0 unknown :-1 s=CLOSED pgs=326 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 >> 172.21.15.194:0/1184248628 conn(0x7f9a940f0b70 msgr2=0x7f9a940f16a0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:17.466 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 shutdown_connections 2024-01-27T23:31:17.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:17.473+0000 7f9a999f3700 1 -- 172.21.15.194:0/1184248628 wait complete. 2024-01-27T23:31:17.467 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:18.836 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:18.836 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:18.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:17.313237+0000 mgr.smithi121.hcppvz (mgr.14162) 186 : cluster [DBG] pgmap v174: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:18.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:17.397072+0000 mon.smithi121 (mon.0) 255 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:31:18.843 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:17.405465+0000 mon.smithi121 (mon.0) 256 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:31:18.843 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:17.472349+0000 mon.smithi121 (mon.0) 257 : audit [DBG] from='client.? 172.21.15.194:0/1184248628' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.944+0000 7f6fabe06700 1 -- 172.21.15.194:0/2576857511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 msgr2=0x7f6fa40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.944+0000 7f6fabe06700 1 --2- 172.21.15.194:0/2576857511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa40f5830 secure :-1 s=READY pgs=327 cs=0 l=1 rev1=1 rx=0x7f6f94009fb0 tx=0x7f6f94009b30).stop 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.944+0000 7f6fabe06700 1 -- 172.21.15.194:0/2576857511 shutdown_connections 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.944+0000 7f6fabe06700 1 --2- 172.21.15.194:0/2576857511 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa40f5830 unknown :-1 s=CLOSED pgs=327 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.944+0000 7f6fabe06700 1 -- 172.21.15.194:0/2576857511 >> 172.21.15.194:0/2576857511 conn(0x7f6fa40f0b70 msgr2=0x7f6fa40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.944+0000 7f6fabe06700 1 -- 172.21.15.194:0/2576857511 shutdown_connections 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.944+0000 7f6fabe06700 1 -- 172.21.15.194:0/2576857511 wait complete. 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.945+0000 7f6fabe06700 1 Processor -- start 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.945+0000 7f6fabe06700 1 -- start start 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.945+0000 7f6fabe06700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa4105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.945+0000 7f6fabe06700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6fa4105b20 con 0x7f6fa40f5410 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.945+0000 7f6fa9ba2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa4105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:19.939 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.945+0000 7f6fa9ba2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa4105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:54634/0 (socket says 172.21.15.194:54634) 2024-01-27T23:31:19.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.945+0000 7f6fa9ba2700 1 -- 172.21.15.194:0/1203898926 learned_addr learned my addr 172.21.15.194:0/1203898926 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:19.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.946+0000 7f6fa9ba2700 1 -- 172.21.15.194:0/1203898926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6f94005040 con 0x7f6fa40f5410 2024-01-27T23:31:19.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.946+0000 7f6fa9ba2700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa4105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f6f94013510 tx=0x7f6f9400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.946+0000 7f6f9a7fc700 1 -- 172.21.15.194:0/1203898926 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f94009320 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.946+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6fa4105d20 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.946+0000 7f6f9a7fc700 1 -- 172.21.15.194:0/1203898926 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6f94013f00 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.946+0000 7f6f9a7fc700 1 -- 172.21.15.194:0/1203898926 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6f94014d90 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.946+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6fa41061b0 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.947+0000 7f6f9a7fc700 1 -- 172.21.15.194:0/1203898926 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6f940263a0 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.947+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6fa4045ad0 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.947+0000 7f6f9a7fc700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6f9002ff60 0x7f6f90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.947+0000 7f6f9a7fc700 1 -- 172.21.15.194:0/1203898926 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6f94041310 con 0x7f6fa40f5410 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.947+0000 7f6fa93a1700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6f9002ff60 0x7f6f90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:19.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.948+0000 7f6fa93a1700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6f9002ff60 0x7f6f90032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f6fa00096a0 tx=0x7f6fa0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:19.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:19.952+0000 7f6f9a7fc700 1 -- 172.21.15.194:0/1203898926 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6f94012920 con 0x7f6fa40f5410 2024-01-27T23:31:20.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.158+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6fa40fae60 con 0x7f6fa40f5410 2024-01-27T23:31:20.152 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.159+0000 7f6f9a7fc700 1 -- 172.21.15.194:0/1203898926 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6f9400f340 con 0x7f6fa40f5410 2024-01-27T23:31:20.153 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:20.153 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6f9002ff60 msgr2=0x7f6f90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6f9002ff60 0x7f6f90032420 secure :-1 s=READY pgs=130 cs=0 l=1 rev1=1 rx=0x7f6fa00096a0 tx=0x7f6fa0006b40).stop 2024-01-27T23:31:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 msgr2=0x7f6fa4105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:20.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa4105630 secure :-1 s=READY pgs=328 cs=0 l=1 rev1=1 rx=0x7f6f94013510 tx=0x7f6f9400b040).stop 2024-01-27T23:31:20.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 shutdown_connections 2024-01-27T23:31:20.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6f9002ff60 0x7f6f90032420 unknown :-1 s=CLOSED pgs=130 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:20.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 --2- 172.21.15.194:0/1203898926 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6fa40f5410 0x7f6fa4105630 unknown :-1 s=CLOSED pgs=328 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:20.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 >> 172.21.15.194:0/1203898926 conn(0x7f6fa40f0b70 msgr2=0x7f6fa40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:20.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 shutdown_connections 2024-01-27T23:31:20.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:20.161+0000 7f6fabe06700 1 -- 172.21.15.194:0/1203898926 wait complete. 2024-01-27T23:31:20.155 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:20.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:19.313443+0000 mgr.smithi121.hcppvz (mgr.14162) 187 : cluster [DBG] pgmap v175: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:20.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:20.160274+0000 mon.smithi121 (mon.0) 258 : audit [DBG] from='client.? 172.21.15.194:0/1203898926' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:21.544 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:21.545 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:22.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:21.313696+0000 mgr.smithi121.hcppvz (mgr.14162) 188 : cluster [DBG] pgmap v176: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.654+0000 7fe943b6a700 1 -- 172.21.15.194:0/3956687931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 msgr2=0x7fe93c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.654+0000 7fe943b6a700 1 --2- 172.21.15.194:0/3956687931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c0f6930 secure :-1 s=READY pgs=329 cs=0 l=1 rev1=1 rx=0x7fe930009fb0 tx=0x7fe930009b30).stop 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.654+0000 7fe943b6a700 1 -- 172.21.15.194:0/3956687931 shutdown_connections 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.654+0000 7fe943b6a700 1 --2- 172.21.15.194:0/3956687931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c0f6930 unknown :-1 s=CLOSED pgs=329 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.654+0000 7fe943b6a700 1 -- 172.21.15.194:0/3956687931 >> 172.21.15.194:0/3956687931 conn(0x7fe93c0f1c70 msgr2=0x7fe93c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.654+0000 7fe943b6a700 1 -- 172.21.15.194:0/3956687931 shutdown_connections 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.654+0000 7fe943b6a700 1 -- 172.21.15.194:0/3956687931 wait complete. 2024-01-27T23:31:22.650 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe943b6a700 1 Processor -- start 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe943b6a700 1 -- start start 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe943b6a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe943b6a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe93c106c20 con 0x7fe93c0f6510 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe941906700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe941906700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:54658/0 (socket says 172.21.15.194:54658) 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe941906700 1 -- 172.21.15.194:0/3948726892 learned_addr learned my addr 172.21.15.194:0/3948726892 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.655+0000 7fe941906700 1 -- 172.21.15.194:0/3948726892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe930005040 con 0x7fe93c0f6510 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.656+0000 7fe941906700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fe930013510 tx=0x7fe93000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.656+0000 7fe92e7fc700 1 -- 172.21.15.194:0/3948726892 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe930009320 con 0x7fe93c0f6510 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.656+0000 7fe92e7fc700 1 -- 172.21.15.194:0/3948726892 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe930013f00 con 0x7fe93c0f6510 2024-01-27T23:31:22.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.656+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe93c106e20 con 0x7fe93c0f6510 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.656+0000 7fe92e7fc700 1 -- 172.21.15.194:0/3948726892 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe930014d90 con 0x7fe93c0f6510 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.656+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe93c1072b0 con 0x7fe93c0f6510 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.656+0000 7fe92e7fc700 1 -- 172.21.15.194:0/3948726892 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe9300095f0 con 0x7fe93c0f6510 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.657+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe93c045ad0 con 0x7fe93c0f6510 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.657+0000 7fe92e7fc700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe92802ff60 0x7fe928032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.657+0000 7fe92e7fc700 1 -- 172.21.15.194:0/3948726892 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe9300410c0 con 0x7fe93c0f6510 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.657+0000 7fe941105700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe92802ff60 0x7fe928032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:22.652 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.657+0000 7fe941105700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe92802ff60 0x7fe928032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe9380096a0 tx=0x7fe938006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:22.655 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.662+0000 7fe92e7fc700 1 -- 172.21.15.194:0/3948726892 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe930014420 con 0x7fe93c0f6510 2024-01-27T23:31:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.870+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe93c0fbf60 con 0x7fe93c0f6510 2024-01-27T23:31:22.864 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.871+0000 7fe92e7fc700 1 -- 172.21.15.194:0/3948726892 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe93001d7f0 con 0x7fe93c0f6510 2024-01-27T23:31:22.865 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:22.865 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe92802ff60 msgr2=0x7fe928032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe92802ff60 0x7fe928032420 secure :-1 s=READY pgs=131 cs=0 l=1 rev1=1 rx=0x7fe9380096a0 tx=0x7fe938006b40).stop 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 msgr2=0x7fe93c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c106730 secure :-1 s=READY pgs=330 cs=0 l=1 rev1=1 rx=0x7fe930013510 tx=0x7fe93000b040).stop 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 shutdown_connections 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe92802ff60 0x7fe928032420 unknown :-1 s=CLOSED pgs=131 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 --2- 172.21.15.194:0/3948726892 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe93c0f6510 0x7fe93c106730 unknown :-1 s=CLOSED pgs=330 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:22.866 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 >> 172.21.15.194:0/3948726892 conn(0x7fe93c0f1c70 msgr2=0x7fe93c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 shutdown_connections 2024-01-27T23:31:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:22.873+0000 7fe943b6a700 1 -- 172.21.15.194:0/3948726892 wait complete. 2024-01-27T23:31:22.867 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:23.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:23 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:22.872518+0000 mon.smithi121 (mon.0) 259 : audit [DBG] from='client.? 172.21.15.194:0/3948726892' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:24.253 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:24.253 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:24.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:23.313974+0000 mgr.smithi121.hcppvz (mgr.14162) 189 : cluster [DBG] pgmap v177: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:25.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.450+0000 7feb8a5fb700 1 -- 172.21.15.194:0/1912224395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 msgr2=0x7feb840f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:25.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.450+0000 7feb8a5fb700 1 --2- 172.21.15.194:0/1912224395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb840f5830 secure :-1 s=READY pgs=331 cs=0 l=1 rev1=1 rx=0x7feb6c009fb0 tx=0x7feb6c009b30).stop 2024-01-27T23:31:25.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.450+0000 7feb8a5fb700 1 -- 172.21.15.194:0/1912224395 shutdown_connections 2024-01-27T23:31:25.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.450+0000 7feb8a5fb700 1 --2- 172.21.15.194:0/1912224395 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb840f5830 unknown :-1 s=CLOSED pgs=331 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:25.444 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.450+0000 7feb8a5fb700 1 -- 172.21.15.194:0/1912224395 >> 172.21.15.194:0/1912224395 conn(0x7feb840f0b70 msgr2=0x7feb840f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:25.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.450+0000 7feb8a5fb700 1 -- 172.21.15.194:0/1912224395 shutdown_connections 2024-01-27T23:31:25.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.450+0000 7feb8a5fb700 1 -- 172.21.15.194:0/1912224395 wait complete. 2024-01-27T23:31:25.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.451+0000 7feb8a5fb700 1 Processor -- start 2024-01-27T23:31:25.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.451+0000 7feb8a5fb700 1 -- start start 2024-01-27T23:31:25.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.451+0000 7feb8a5fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb8410b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:25.445 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.451+0000 7feb8a5fb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7feb84107240 con 0x7feb840f5410 2024-01-27T23:31:25.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.451+0000 7feb83fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb8410b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:25.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.451+0000 7feb83fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb8410b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45970/0 (socket says 172.21.15.194:45970) 2024-01-27T23:31:25.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.451+0000 7feb83fff700 1 -- 172.21.15.194:0/3397247922 learned_addr learned my addr 172.21.15.194:0/3397247922 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:25.446 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.452+0000 7feb83fff700 1 -- 172.21.15.194:0/3397247922 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7feb6c005040 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.452+0000 7feb83fff700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb8410b850 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7feb6c013510 tx=0x7feb6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.452+0000 7feb80ff9700 1 -- 172.21.15.194:0/3397247922 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c009320 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.452+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7feb84060220 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.452+0000 7feb80ff9700 1 -- 172.21.15.194:0/3397247922 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7feb6c013f00 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.452+0000 7feb80ff9700 1 -- 172.21.15.194:0/3397247922 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7feb6c014d90 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.452+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7feb8410bd40 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.453+0000 7feb80ff9700 1 -- 172.21.15.194:0/3397247922 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7feb6c014420 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.453+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7feb84045ad0 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.453+0000 7feb80ff9700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7feb7002ff60 0x7feb70032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.453+0000 7feb80ff9700 1 -- 172.21.15.194:0/3397247922 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7feb6c040ff0 con 0x7feb840f5410 2024-01-27T23:31:25.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.453+0000 7feb837fe700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7feb7002ff60 0x7feb70032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:25.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.454+0000 7feb837fe700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7feb7002ff60 0x7feb70032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7feb740096a0 tx=0x7feb74006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:25.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.458+0000 7feb80ff9700 1 -- 172.21.15.194:0/3397247922 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7feb6c0095f0 con 0x7feb840f5410 2024-01-27T23:31:25.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.664+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7feb840fae60 con 0x7feb840f5410 2024-01-27T23:31:25.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.665+0000 7feb80ff9700 1 -- 172.21.15.194:0/3397247922 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7feb6c01db70 con 0x7feb840f5410 2024-01-27T23:31:25.658 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:25.659 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:25.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7feb7002ff60 msgr2=0x7feb70032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:25.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7feb7002ff60 0x7feb70032420 secure :-1 s=READY pgs=132 cs=0 l=1 rev1=1 rx=0x7feb740096a0 tx=0x7feb74006b40).stop 2024-01-27T23:31:25.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 msgr2=0x7feb8410b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb8410b850 secure :-1 s=READY pgs=332 cs=0 l=1 rev1=1 rx=0x7feb6c013510 tx=0x7feb6c00b040).stop 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 shutdown_connections 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7feb7002ff60 0x7feb70032420 unknown :-1 s=CLOSED pgs=132 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 --2- 172.21.15.194:0/3397247922 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7feb840f5410 0x7feb8410b850 unknown :-1 s=CLOSED pgs=332 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 >> 172.21.15.194:0/3397247922 conn(0x7feb840f0b70 msgr2=0x7feb840f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 shutdown_connections 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:25.667+0000 7feb8a5fb700 1 -- 172.21.15.194:0/3397247922 wait complete. 2024-01-27T23:31:25.661 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:25.314284+0000 mgr.smithi121.hcppvz (mgr.14162) 190 : cluster [DBG] pgmap v178: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:26.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:25.666351+0000 mon.smithi121 (mon.0) 260 : audit [DBG] from='client.? 172.21.15.194:0/3397247922' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:27.055 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:27.056 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 -- 172.21.15.194:0/3663219611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 msgr2=0x7f7aac0f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 --2- 172.21.15.194:0/3663219611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac0f55c0 secure :-1 s=READY pgs=333 cs=0 l=1 rev1=1 rx=0x7f7a9c009fb0 tx=0x7f7a9c009b30).stop 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 -- 172.21.15.194:0/3663219611 shutdown_connections 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 --2- 172.21.15.194:0/3663219611 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac0f55c0 unknown :-1 s=CLOSED pgs=333 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 -- 172.21.15.194:0/3663219611 >> 172.21.15.194:0/3663219611 conn(0x7f7aac0f09a0 msgr2=0x7f7aac0f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 -- 172.21.15.194:0/3663219611 shutdown_connections 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 -- 172.21.15.194:0/3663219611 wait complete. 2024-01-27T23:31:28.221 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 Processor -- start 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.226+0000 7f7ab25dc700 1 -- start start 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab25dc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac1064d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab25dc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7aac1069c0 con 0x7f7aac0f51a0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab15da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac1064d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab15da700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac1064d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45980/0 (socket says 172.21.15.194:45980) 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab15da700 1 -- 172.21.15.194:0/2949687807 learned_addr learned my addr 172.21.15.194:0/2949687807 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab15da700 1 -- 172.21.15.194:0/2949687807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7a9c005040 con 0x7f7aac0f51a0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab15da700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac1064d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7a9c013510 tx=0x7f7a9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7aa1ffb700 1 -- 172.21.15.194:0/2949687807 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a9c009320 con 0x7f7aac0f51a0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.227+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7aac106bc0 con 0x7f7aac0f51a0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.228+0000 7f7aa1ffb700 1 -- 172.21.15.194:0/2949687807 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7a9c013f00 con 0x7f7aac0f51a0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.228+0000 7f7aa1ffb700 1 -- 172.21.15.194:0/2949687807 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7a9c014dc0 con 0x7f7aac0f51a0 2024-01-27T23:31:28.222 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.228+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f7aac107050 con 0x7f7aac0f51a0 2024-01-27T23:31:28.223 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.228+0000 7f7aa1ffb700 1 -- 172.21.15.194:0/2949687807 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f7a9c01d3d0 con 0x7f7aac0f51a0 2024-01-27T23:31:28.223 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.228+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7aac045ad0 con 0x7f7aac0f51a0 2024-01-27T23:31:28.223 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.229+0000 7f7aa1ffb700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7a9802ff60 0x7f7a98032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:28.224 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.229+0000 7f7aa1ffb700 1 -- 172.21.15.194:0/2949687807 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f7a9c041040 con 0x7f7aac0f51a0 2024-01-27T23:31:28.224 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.229+0000 7f7ab0dd9700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7a9802ff60 0x7f7a98032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:28.224 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.229+0000 7f7ab0dd9700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7a9802ff60 0x7f7a98032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7aa80096a0 tx=0x7f7aa8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:28.227 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.233+0000 7f7aa1ffb700 1 -- 172.21.15.194:0/2949687807 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7a9c0126f0 con 0x7f7aac0f51a0 2024-01-27T23:31:28.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.439+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7aac107f00 con 0x7f7aac0f51a0 2024-01-27T23:31:28.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.440+0000 7f7aa1ffb700 1 -- 172.21.15.194:0/2949687807 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f7a9c012330 con 0x7f7aac0f51a0 2024-01-27T23:31:28.433 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:28.434 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7a9802ff60 msgr2=0x7f7a98032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7a9802ff60 0x7f7a98032420 secure :-1 s=READY pgs=133 cs=0 l=1 rev1=1 rx=0x7f7aa80096a0 tx=0x7f7aa8006b40).stop 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 msgr2=0x7f7aac1064d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac1064d0 secure :-1 s=READY pgs=334 cs=0 l=1 rev1=1 rx=0x7f7a9c013510 tx=0x7f7a9c00b040).stop 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 shutdown_connections 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f7a9802ff60 0x7f7a98032420 unknown :-1 s=CLOSED pgs=133 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 --2- 172.21.15.194:0/2949687807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f7aac0f51a0 0x7f7aac1064d0 unknown :-1 s=CLOSED pgs=334 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 >> 172.21.15.194:0/2949687807 conn(0x7f7aac0f09a0 msgr2=0x7f7aac05fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 shutdown_connections 2024-01-27T23:31:28.436 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:28.442+0000 7f7ab25dc700 1 -- 172.21.15.194:0/2949687807 wait complete. 2024-01-27T23:31:28.437 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:28.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:27.314533+0000 mgr.smithi121.hcppvz (mgr.14162) 191 : cluster [DBG] pgmap v179: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:29.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:29 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:28.441212+0000 mon.smithi121 (mon.0) 261 : audit [DBG] from='client.? 172.21.15.194:0/2949687807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:29.818 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:29.818 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:30.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:29.314855+0000 mgr.smithi121.hcppvz (mgr.14162) 192 : cluster [DBG] pgmap v180: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:30.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.949+0000 7f0915087700 1 -- 172.21.15.194:0/947415426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 msgr2=0x7f09100f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:30.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.949+0000 7f0915087700 1 --2- 172.21.15.194:0/947415426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f09100f6930 secure :-1 s=READY pgs=335 cs=0 l=1 rev1=1 rx=0x7f08f8009fb0 tx=0x7f08f8009b30).stop 2024-01-27T23:31:30.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.949+0000 7f0915087700 1 -- 172.21.15.194:0/947415426 shutdown_connections 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.949+0000 7f0915087700 1 --2- 172.21.15.194:0/947415426 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f09100f6930 unknown :-1 s=CLOSED pgs=335 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.949+0000 7f0915087700 1 -- 172.21.15.194:0/947415426 >> 172.21.15.194:0/947415426 conn(0x7f09100f1c70 msgr2=0x7f09100f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.949+0000 7f0915087700 1 -- 172.21.15.194:0/947415426 shutdown_connections 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.949+0000 7f0915087700 1 -- 172.21.15.194:0/947415426 wait complete. 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f0915087700 1 Processor -- start 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f0915087700 1 -- start start 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f0915087700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f0910106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f0915087700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0910106c20 con 0x7f09100f6510 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f090ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f0910106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f090ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f0910106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:45996/0 (socket says 172.21.15.194:45996) 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f090ed9d700 1 -- 172.21.15.194:0/4136133846 learned_addr learned my addr 172.21.15.194:0/4136133846 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:30.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f090ed9d700 1 -- 172.21.15.194:0/4136133846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f08f8005040 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f090ed9d700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f0910106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f08f8013510 tx=0x7f08f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.950+0000 7f09077fe700 1 -- 172.21.15.194:0/4136133846 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08f8009320 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.951+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0910106e20 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.951+0000 7f09077fe700 1 -- 172.21.15.194:0/4136133846 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f08f8013f00 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.951+0000 7f09077fe700 1 -- 172.21.15.194:0/4136133846 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f08f8014d90 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.951+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f09101072b0 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.951+0000 7f09077fe700 1 -- 172.21.15.194:0/4136133846 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f08f8014420 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.951+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0910045ad0 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.951+0000 7f09077fe700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f08fc02ff60 0x7f08fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.952+0000 7f09077fe700 1 -- 172.21.15.194:0/4136133846 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f08f80410c0 con 0x7f09100f6510 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.952+0000 7f090e59c700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f08fc02ff60 0x7f08fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:30.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.952+0000 7f090e59c700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f08fc02ff60 0x7f08fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f09000096a0 tx=0x7f0900006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:30.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:30.956+0000 7f09077fe700 1 -- 172.21.15.194:0/4136133846 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f08f80095f0 con 0x7f09100f6510 2024-01-27T23:31:31.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.166+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f09100fbf60 con 0x7f09100f6510 2024-01-27T23:31:31.160 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.166+0000 7f09077fe700 1 -- 172.21.15.194:0/4136133846 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f08f801db70 con 0x7f09100f6510 2024-01-27T23:31:31.160 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:31.160 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:31.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f08fc02ff60 msgr2=0x7f08fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f08fc02ff60 0x7f08fc032420 secure :-1 s=READY pgs=134 cs=0 l=1 rev1=1 rx=0x7f09000096a0 tx=0x7f0900006b40).stop 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 msgr2=0x7f0910106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f0910106730 secure :-1 s=READY pgs=336 cs=0 l=1 rev1=1 rx=0x7f08f8013510 tx=0x7f08f800b040).stop 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 shutdown_connections 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f08fc02ff60 0x7f08fc032420 unknown :-1 s=CLOSED pgs=134 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 --2- 172.21.15.194:0/4136133846 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f09100f6510 0x7f0910106730 unknown :-1 s=CLOSED pgs=336 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 >> 172.21.15.194:0/4136133846 conn(0x7f09100f1c70 msgr2=0x7f09100f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 shutdown_connections 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:31.168+0000 7f0915087700 1 -- 172.21.15.194:0/4136133846 wait complete. 2024-01-27T23:31:31.162 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:31.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:31 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:31.167811+0000 mon.smithi121 (mon.0) 262 : audit [DBG] from='client.? 172.21.15.194:0/4136133846' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:32.543 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:32.543 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:32.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:31.315126+0000 mgr.smithi121.hcppvz (mgr.14162) 193 : cluster [DBG] pgmap v181: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:33.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.716+0000 7fe225fba700 1 -- 172.21.15.194:0/2433311415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 msgr2=0x7fe2200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:33.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.716+0000 7fe225fba700 1 --2- 172.21.15.194:0/2433311415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe2200f6930 secure :-1 s=READY pgs=337 cs=0 l=1 rev1=1 rx=0x7fe210009fb0 tx=0x7fe210009b30).stop 2024-01-27T23:31:33.710 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.716+0000 7fe225fba700 1 -- 172.21.15.194:0/2433311415 shutdown_connections 2024-01-27T23:31:33.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.716+0000 7fe225fba700 1 --2- 172.21.15.194:0/2433311415 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe2200f6930 unknown :-1 s=CLOSED pgs=337 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:33.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.716+0000 7fe225fba700 1 -- 172.21.15.194:0/2433311415 >> 172.21.15.194:0/2433311415 conn(0x7fe2200f1c70 msgr2=0x7fe2200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:33.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.716+0000 7fe225fba700 1 -- 172.21.15.194:0/2433311415 shutdown_connections 2024-01-27T23:31:33.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.717+0000 7fe225fba700 1 -- 172.21.15.194:0/2433311415 wait complete. 2024-01-27T23:31:33.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.717+0000 7fe225fba700 1 Processor -- start 2024-01-27T23:31:33.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.717+0000 7fe225fba700 1 -- start start 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.717+0000 7fe225fba700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe220106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.717+0000 7fe225fba700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe220106c20 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.717+0000 7fe21f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe220106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe21f7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe220106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:46012/0 (socket says 172.21.15.194:46012) 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe21f7fe700 1 -- 172.21.15.194:0/2525170170 learned_addr learned my addr 172.21.15.194:0/2525170170 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe21f7fe700 1 -- 172.21.15.194:0/2525170170 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe210005040 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe21f7fe700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe220106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fe210013510 tx=0x7fe21000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe207fff700 1 -- 172.21.15.194:0/2525170170 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe210009320 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe220106e20 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe207fff700 1 -- 172.21.15.194:0/2525170170 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe210013f00 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe207fff700 1 -- 172.21.15.194:0/2525170170 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe210014d90 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.718+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe2201072b0 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.719+0000 7fe207fff700 1 -- 172.21.15.194:0/2525170170 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe2100263a0 con 0x7fe2200f6510 2024-01-27T23:31:33.713 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.719+0000 7fe207fff700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe20802ff60 0x7fe208032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:33.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.719+0000 7fe207fff700 1 -- 172.21.15.194:0/2525170170 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe210041010 con 0x7fe2200f6510 2024-01-27T23:31:33.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.719+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe220045ad0 con 0x7fe2200f6510 2024-01-27T23:31:33.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.719+0000 7fe21effd700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe20802ff60 0x7fe208032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:33.714 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.720+0000 7fe21effd700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe20802ff60 0x7fe208032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:33.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.724+0000 7fe207fff700 1 -- 172.21.15.194:0/2525170170 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe210012920 con 0x7fe2200f6510 2024-01-27T23:31:33.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.929+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe220107ff0 con 0x7fe2200f6510 2024-01-27T23:31:33.923 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.929+0000 7fe207fff700 1 -- 172.21.15.194:0/2525170170 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe21002b020 con 0x7fe2200f6510 2024-01-27T23:31:33.923 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:33.923 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:33.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe20802ff60 msgr2=0x7fe208032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:33.924 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe20802ff60 0x7fe208032420 secure :-1 s=READY pgs=135 cs=0 l=1 rev1=1 rx=0x7fe2140096a0 tx=0x7fe214006b40).stop 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 msgr2=0x7fe220106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe220106730 secure :-1 s=READY pgs=338 cs=0 l=1 rev1=1 rx=0x7fe210013510 tx=0x7fe21000b040).stop 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 shutdown_connections 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe20802ff60 0x7fe208032420 unknown :-1 s=CLOSED pgs=135 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 --2- 172.21.15.194:0/2525170170 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe2200f6510 0x7fe220106730 unknown :-1 s=CLOSED pgs=338 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 >> 172.21.15.194:0/2525170170 conn(0x7fe2200f1c70 msgr2=0x7fe2200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 shutdown_connections 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:33.931+0000 7fe225fba700 1 -- 172.21.15.194:0/2525170170 wait complete. 2024-01-27T23:31:33.925 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:34.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:33.315473+0000 mgr.smithi121.hcppvz (mgr.14162) 194 : cluster [DBG] pgmap v182: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:34.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:33.930711+0000 mon.smithi121 (mon.0) 263 : audit [DBG] from='client.? 172.21.15.194:0/2525170170' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:35.327 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:35.327 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3523178314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 msgr2=0x7f19b80f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 --2- 172.21.15.194:0/3523178314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b80f6930 secure :-1 s=READY pgs=339 cs=0 l=1 rev1=1 rx=0x7f19b4009fb0 tx=0x7f19b4009b30).stop 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3523178314 shutdown_connections 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 --2- 172.21.15.194:0/3523178314 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b80f6930 unknown :-1 s=CLOSED pgs=339 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3523178314 >> 172.21.15.194:0/3523178314 conn(0x7f19b80f1c70 msgr2=0x7f19b80f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3523178314 shutdown_connections 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3523178314 wait complete. 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 Processor -- start 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.441+0000 7f19bf2d2700 1 -- start start 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19bf2d2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b806b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:36.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19bf2d2700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f19b8069e20 con 0x7f19b80f6510 2024-01-27T23:31:36.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19bd06e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b806b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:36.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19bd06e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b806b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37276/0 (socket says 172.21.15.194:37276) 2024-01-27T23:31:36.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19bd06e700 1 -- 172.21.15.194:0/3364874986 learned_addr learned my addr 172.21.15.194:0/3364874986 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:36.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19bd06e700 1 -- 172.21.15.194:0/3364874986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f19b4005040 con 0x7f19b80f6510 2024-01-27T23:31:36.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19bd06e700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b806b9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f19b4013510 tx=0x7f19b400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:36.437 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.442+0000 7f19adffb700 1 -- 172.21.15.194:0/3364874986 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b4009320 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.443+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f19b8069fc0 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.443+0000 7f19adffb700 1 -- 172.21.15.194:0/3364874986 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f19b4013f00 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.443+0000 7f19adffb700 1 -- 172.21.15.194:0/3364874986 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f19b4014d90 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.443+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19b806a450 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.443+0000 7f19adffb700 1 -- 172.21.15.194:0/3364874986 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19b40095f0 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.444+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f19b8059100 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.444+0000 7f19adffb700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f19a402ff60 0x7f19a4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.444+0000 7f19adffb700 1 -- 172.21.15.194:0/3364874986 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f19b4041000 con 0x7f19b80f6510 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.444+0000 7f19bc86d700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f19a402ff60 0x7f19a4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:36.438 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.444+0000 7f19bc86d700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f19a402ff60 0x7f19a4032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f19a80096a0 tx=0x7f19a8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:36.442 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.448+0000 7f19adffb700 1 -- 172.21.15.194:0/3364874986 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f19b401d7f0 con 0x7f19b80f6510 2024-01-27T23:31:36.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:35.315854+0000 mgr.smithi121.hcppvz (mgr.14162) 195 : cluster [DBG] pgmap v183: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:36.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.654+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19b80fbf60 con 0x7f19b80f6510 2024-01-27T23:31:36.648 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.655+0000 7f19adffb700 1 -- 172.21.15.194:0/3364874986 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f19b4010230 con 0x7f19b80f6510 2024-01-27T23:31:36.649 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:36.649 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f19a402ff60 msgr2=0x7f19a4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f19a402ff60 0x7f19a4032420 secure :-1 s=READY pgs=136 cs=0 l=1 rev1=1 rx=0x7f19a80096a0 tx=0x7f19a8006b40).stop 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 msgr2=0x7f19b806b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b806b9a0 secure :-1 s=READY pgs=340 cs=0 l=1 rev1=1 rx=0x7f19b4013510 tx=0x7f19b400b040).stop 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 shutdown_connections 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f19a402ff60 0x7f19a4032420 unknown :-1 s=CLOSED pgs=136 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 --2- 172.21.15.194:0/3364874986 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19b80f6510 0x7f19b806b9a0 unknown :-1 s=CLOSED pgs=340 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 >> 172.21.15.194:0/3364874986 conn(0x7f19b80f1c70 msgr2=0x7f19b80f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 shutdown_connections 2024-01-27T23:31:36.651 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:36.657+0000 7f19bf2d2700 1 -- 172.21.15.194:0/3364874986 wait complete. 2024-01-27T23:31:36.652 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:37.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:37 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:36.656439+0000 mon.smithi121 (mon.0) 264 : audit [DBG] from='client.? 172.21.15.194:0/3364874986' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:38.031 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:38.031 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:38.590 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:37.316240+0000 mgr.smithi121.hcppvz (mgr.14162) 196 : cluster [DBG] pgmap v184: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.214+0000 7f3451f32700 1 -- 172.21.15.194:0/1211450556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 msgr2=0x7f344c0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.214+0000 7f3451f32700 1 --2- 172.21.15.194:0/1211450556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c0f66c0 secure :-1 s=READY pgs=341 cs=0 l=1 rev1=1 rx=0x7f3440009fb0 tx=0x7f3440009b30).stop 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.215+0000 7f3451f32700 1 -- 172.21.15.194:0/1211450556 shutdown_connections 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.215+0000 7f3451f32700 1 --2- 172.21.15.194:0/1211450556 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c0f66c0 unknown :-1 s=CLOSED pgs=341 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.215+0000 7f3451f32700 1 -- 172.21.15.194:0/1211450556 >> 172.21.15.194:0/1211450556 conn(0x7f344c0f1aa0 msgr2=0x7f344c0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.215+0000 7f3451f32700 1 -- 172.21.15.194:0/1211450556 shutdown_connections 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.215+0000 7f3451f32700 1 -- 172.21.15.194:0/1211450556 wait complete. 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.215+0000 7f3451f32700 1 Processor -- start 2024-01-27T23:31:39.209 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.215+0000 7f3451f32700 1 -- start start 2024-01-27T23:31:39.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3451f32700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c1064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:39.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3451f32700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f344c1069a0 con 0x7f344c0f62a0 2024-01-27T23:31:39.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3450f30700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c1064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:39.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3450f30700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c1064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37286/0 (socket says 172.21.15.194:37286) 2024-01-27T23:31:39.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3450f30700 1 -- 172.21.15.194:0/763757534 learned_addr learned my addr 172.21.15.194:0/763757534 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:39.211 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3450f30700 1 -- 172.21.15.194:0/763757534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3440005040 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3450f30700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c1064b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f3440013510 tx=0x7f344000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f34497fa700 1 -- 172.21.15.194:0/763757534 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3440009320 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f344c106ba0 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f34497fa700 1 -- 172.21.15.194:0/763757534 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3440013f00 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.216+0000 7f34497fa700 1 -- 172.21.15.194:0/763757534 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3440014dc0 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.217+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f344c107030 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.217+0000 7f34497fa700 1 -- 172.21.15.194:0/763757534 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f3440014420 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.217+0000 7f34497fa700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f343c02ff60 0x7f343c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.217+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f344c045ad0 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.217+0000 7f34497fa700 1 -- 172.21.15.194:0/763757534 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f3440041110 con 0x7f344c0f62a0 2024-01-27T23:31:39.212 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.218+0000 7f344bfff700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f343c02ff60 0x7f343c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:39.213 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.218+0000 7f344bfff700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f343c02ff60 0x7f343c032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f34380096a0 tx=0x7f3438006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:39.215 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.222+0000 7f34497fa700 1 -- 172.21.15.194:0/763757534 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f34400095f0 con 0x7f344c0f62a0 2024-01-27T23:31:39.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.430+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f344c107ee0 con 0x7f344c0f62a0 2024-01-27T23:31:39.424 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.430+0000 7f34497fa700 1 -- 172.21.15.194:0/763757534 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f344001cb10 con 0x7f344c0f62a0 2024-01-27T23:31:39.424 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:39.424 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f343c02ff60 msgr2=0x7f343c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f343c02ff60 0x7f343c032420 secure :-1 s=READY pgs=137 cs=0 l=1 rev1=1 rx=0x7f34380096a0 tx=0x7f3438006b40).stop 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 msgr2=0x7f344c1064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c1064b0 secure :-1 s=READY pgs=342 cs=0 l=1 rev1=1 rx=0x7f3440013510 tx=0x7f344000b040).stop 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 shutdown_connections 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f343c02ff60 0x7f343c032420 unknown :-1 s=CLOSED pgs=137 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 --2- 172.21.15.194:0/763757534 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f344c0f62a0 0x7f344c1064b0 unknown :-1 s=CLOSED pgs=342 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 >> 172.21.15.194:0/763757534 conn(0x7f344c0f1aa0 msgr2=0x7f344c0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 shutdown_connections 2024-01-27T23:31:39.426 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:39.432+0000 7f3451f32700 1 -- 172.21.15.194:0/763757534 wait complete. 2024-01-27T23:31:39.427 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:40.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:39.316591+0000 mgr.smithi121.hcppvz (mgr.14162) 197 : cluster [DBG] pgmap v185: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:40.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:39.431915+0000 mon.smithi121 (mon.0) 265 : audit [DBG] from='client.? 172.21.15.194:0/763757534' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:40.811 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:40.812 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:42.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.032+0000 7f9d06e9c700 1 -- 172.21.15.194:0/1900327850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 msgr2=0x7f9d000f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:42.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.032+0000 7f9d06e9c700 1 --2- 172.21.15.194:0/1900327850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d000f55c0 secure :-1 s=READY pgs=343 cs=0 l=1 rev1=1 rx=0x7f9cf0009fb0 tx=0x7f9cf0009b30).stop 2024-01-27T23:31:42.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.032+0000 7f9d06e9c700 1 -- 172.21.15.194:0/1900327850 shutdown_connections 2024-01-27T23:31:42.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.032+0000 7f9d06e9c700 1 --2- 172.21.15.194:0/1900327850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d000f55c0 unknown :-1 s=CLOSED pgs=343 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:42.027 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.032+0000 7f9d06e9c700 1 -- 172.21.15.194:0/1900327850 >> 172.21.15.194:0/1900327850 conn(0x7f9d000f09a0 msgr2=0x7f9d000f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.032+0000 7f9d06e9c700 1 -- 172.21.15.194:0/1900327850 shutdown_connections 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.032+0000 7f9d06e9c700 1 -- 172.21.15.194:0/1900327850 wait complete. 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d06e9c700 1 Processor -- start 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d06e9c700 1 -- start start 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d06e9c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d001091d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d06e9c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9d00060220 con 0x7f9d000f51a0 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d05e9a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d001091d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d05e9a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d001091d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37310/0 (socket says 172.21.15.194:37310) 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d05e9a700 1 -- 172.21.15.194:0/2064735708 learned_addr learned my addr 172.21.15.194:0/2064735708 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d05e9a700 1 -- 172.21.15.194:0/2064735708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9cf0005040 con 0x7f9d000f51a0 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.033+0000 7f9d05e9a700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d001091d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f9cf0013510 tx=0x7f9cf000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.034+0000 7f9cf67fc700 1 -- 172.21.15.194:0/2064735708 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf0009320 con 0x7f9d000f51a0 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.034+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9d00060420 con 0x7f9d000f51a0 2024-01-27T23:31:42.028 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.034+0000 7f9cf67fc700 1 -- 172.21.15.194:0/2064735708 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9cf0013f00 con 0x7f9d000f51a0 2024-01-27T23:31:42.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.034+0000 7f9cf67fc700 1 -- 172.21.15.194:0/2064735708 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9cf0014dc0 con 0x7f9d000f51a0 2024-01-27T23:31:42.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.034+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9d00109880 con 0x7f9d000f51a0 2024-01-27T23:31:42.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.034+0000 7f9cf67fc700 1 -- 172.21.15.194:0/2064735708 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9cf00095f0 con 0x7f9d000f51a0 2024-01-27T23:31:42.029 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.035+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9d00059100 con 0x7f9d000f51a0 2024-01-27T23:31:42.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.035+0000 7f9cf67fc700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9cec02ff60 0x7f9cec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:42.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.035+0000 7f9cf67fc700 1 -- 172.21.15.194:0/2064735708 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9cf00410c0 con 0x7f9d000f51a0 2024-01-27T23:31:42.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.035+0000 7f9d05699700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9cec02ff60 0x7f9cec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:42.030 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.035+0000 7f9d05699700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9cec02ff60 0x7f9cec032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:42.033 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.039+0000 7f9cf67fc700 1 -- 172.21.15.194:0/2064735708 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9cf0014420 con 0x7f9d000f51a0 2024-01-27T23:31:42.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.242+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9d00045ad0 con 0x7f9d000f51a0 2024-01-27T23:31:42.236 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.243+0000 7f9cf67fc700 1 -- 172.21.15.194:0/2064735708 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9cf0010230 con 0x7f9d000f51a0 2024-01-27T23:31:42.237 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:42.237 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:42.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9cec02ff60 msgr2=0x7f9cec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9cec02ff60 0x7f9cec032420 secure :-1 s=READY pgs=138 cs=0 l=1 rev1=1 rx=0x7f9cfc0096a0 tx=0x7f9cfc006b40).stop 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 msgr2=0x7f9d001091d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d001091d0 secure :-1 s=READY pgs=344 cs=0 l=1 rev1=1 rx=0x7f9cf0013510 tx=0x7f9cf000b040).stop 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 shutdown_connections 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9cec02ff60 0x7f9cec032420 unknown :-1 s=CLOSED pgs=138 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 --2- 172.21.15.194:0/2064735708 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9d000f51a0 0x7f9d001091d0 unknown :-1 s=CLOSED pgs=344 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 >> 172.21.15.194:0/2064735708 conn(0x7f9d000f09a0 msgr2=0x7f9d000f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 shutdown_connections 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:42.245+0000 7f9d06e9c700 1 -- 172.21.15.194:0/2064735708 wait complete. 2024-01-27T23:31:42.239 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:42.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:41.316829+0000 mgr.smithi121.hcppvz (mgr.14162) 198 : cluster [DBG] pgmap v186: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:42.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:42.244155+0000 mon.smithi121 (mon.0) 266 : audit [DBG] from='client.? 172.21.15.194:0/2064735708' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:43.642 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:43.643 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:44.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.722+0000 7fe665068700 1 -- 172.21.15.194:0/62039661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6600f6a10 msgr2=0x7fe6600f6e30 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:44.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.722+0000 7fe665068700 1 --2- 172.21.15.194:0/62039661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6600f6a10 0x7fe6600f6e30 secure :-1 s=READY pgs=345 cs=0 l=1 rev1=1 rx=0x7fe654009fb0 tx=0x7fe654009b30).stop 2024-01-27T23:31:44.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.722+0000 7fe665068700 1 -- 172.21.15.194:0/62039661 shutdown_connections 2024-01-27T23:31:44.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.722+0000 7fe665068700 1 --2- 172.21.15.194:0/62039661 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6600f6a10 0x7fe6600f6e30 unknown :-1 s=CLOSED pgs=345 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:44.717 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.722+0000 7fe665068700 1 -- 172.21.15.194:0/62039661 >> 172.21.15.194:0/62039661 conn(0x7fe6600f02e0 msgr2=0x7fe6600f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.722+0000 7fe665068700 1 -- 172.21.15.194:0/62039661 shutdown_connections 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.722+0000 7fe665068700 1 -- 172.21.15.194:0/62039661 wait complete. 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe665068700 1 Processor -- start 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe665068700 1 -- start start 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe665068700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6601056f0 0x7fe660105b10 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe665068700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fe660106000 con 0x7fe6601056f0 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe65ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6601056f0 0x7fe660105b10 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe65ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6601056f0 0x7fe660105b10 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:37336/0 (socket says 172.21.15.194:37336) 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe65ed9d700 1 -- 172.21.15.194:0/511804934 learned_addr learned my addr 172.21.15.194:0/511804934 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe65ed9d700 1 -- 172.21.15.194:0/511804934 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fe654005040 con 0x7fe6601056f0 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.723+0000 7fe65ed9d700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6601056f0 0x7fe660105b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe654009f80 tx=0x7fe654009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.724+0000 7fe6477fe700 1 -- 172.21.15.194:0/511804934 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe6540136c0 con 0x7fe6601056f0 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.724+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fe660106200 con 0x7fe6601056f0 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.724+0000 7fe6477fe700 1 -- 172.21.15.194:0/511804934 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fe654013e70 con 0x7fe6601056f0 2024-01-27T23:31:44.718 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.724+0000 7fe6477fe700 1 -- 172.21.15.194:0/511804934 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fe654014420 con 0x7fe6601056f0 2024-01-27T23:31:44.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.724+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fe660106d50 con 0x7fe6601056f0 2024-01-27T23:31:44.719 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.724+0000 7fe6477fe700 1 -- 172.21.15.194:0/511804934 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fe654025060 con 0x7fe6601056f0 2024-01-27T23:31:44.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.725+0000 7fe6477fe700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe64802ff60 0x7fe648032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:44.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.725+0000 7fe6477fe700 1 -- 172.21.15.194:0/511804934 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fe654040e50 con 0x7fe6601056f0 2024-01-27T23:31:44.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.725+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fe660059100 con 0x7fe6601056f0 2024-01-27T23:31:44.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.725+0000 7fe65e59c700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe64802ff60 0x7fe648032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:44.720 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.725+0000 7fe65e59c700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe64802ff60 0x7fe648032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fe6500096a0 tx=0x7fe650006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:44.723 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.729+0000 7fe6477fe700 1 -- 172.21.15.194:0/511804934 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fe654028440 con 0x7fe6601056f0 2024-01-27T23:31:44.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:43.317135+0000 mgr.smithi121.hcppvz (mgr.14162) 199 : cluster [DBG] pgmap v187: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:44.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.935+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fe660107450 con 0x7fe6601056f0 2024-01-27T23:31:44.929 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.936+0000 7fe6477fe700 1 -- 172.21.15.194:0/511804934 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fe65401d3a0 con 0x7fe6601056f0 2024-01-27T23:31:44.929 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:44.930 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:44.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe64802ff60 msgr2=0x7fe648032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:44.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe64802ff60 0x7fe648032420 secure :-1 s=READY pgs=139 cs=0 l=1 rev1=1 rx=0x7fe6500096a0 tx=0x7fe650006b40).stop 2024-01-27T23:31:44.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6601056f0 msgr2=0x7fe660105b10 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:44.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6601056f0 0x7fe660105b10 secure :-1 s=READY pgs=346 cs=0 l=1 rev1=1 rx=0x7fe654009f80 tx=0x7fe654009b30).stop 2024-01-27T23:31:44.931 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 shutdown_connections 2024-01-27T23:31:44.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fe64802ff60 0x7fe648032420 unknown :-1 s=CLOSED pgs=139 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:44.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 --2- 172.21.15.194:0/511804934 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fe6601056f0 0x7fe660105b10 unknown :-1 s=CLOSED pgs=346 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:44.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 >> 172.21.15.194:0/511804934 conn(0x7fe6600f02e0 msgr2=0x7fe6600f0e70 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:44.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 shutdown_connections 2024-01-27T23:31:44.932 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:44.938+0000 7fe665068700 1 -- 172.21.15.194:0/511804934 wait complete. 2024-01-27T23:31:44.932 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:45.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:45 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:44.936996+0000 mon.smithi121 (mon.0) 267 : audit [DBG] from='client.? 172.21.15.194:0/511804934' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:46.314 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:46.314 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:46.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:45.317502+0000 mgr.smithi121.hcppvz (mgr.14162) 200 : cluster [DBG] pgmap v188: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:47.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.517+0000 7fd282117700 1 -- 172.21.15.194:0/251637424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 msgr2=0x7fd27c0f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:47.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.517+0000 7fd282117700 1 --2- 172.21.15.194:0/251637424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c0f90f0 secure :-1 s=READY pgs=347 cs=0 l=1 rev1=1 rx=0x7fd264009fb0 tx=0x7fd264009b30).stop 2024-01-27T23:31:47.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.517+0000 7fd282117700 1 -- 172.21.15.194:0/251637424 shutdown_connections 2024-01-27T23:31:47.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.517+0000 7fd282117700 1 --2- 172.21.15.194:0/251637424 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c0f90f0 unknown :-1 s=CLOSED pgs=347 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:47.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.517+0000 7fd282117700 1 -- 172.21.15.194:0/251637424 >> 172.21.15.194:0/251637424 conn(0x7fd27c0f0b70 msgr2=0x7fd27c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:47.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.517+0000 7fd282117700 1 -- 172.21.15.194:0/251637424 shutdown_connections 2024-01-27T23:31:47.511 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.517+0000 7fd282117700 1 -- 172.21.15.194:0/251637424 wait complete. 2024-01-27T23:31:47.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.518+0000 7fd282117700 1 Processor -- start 2024-01-27T23:31:47.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.518+0000 7fd282117700 1 -- start start 2024-01-27T23:31:47.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.518+0000 7fd282117700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c1066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:47.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.518+0000 7fd282117700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd27c106bd0 con 0x7fd27c060280 2024-01-27T23:31:47.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd27b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c1066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:47.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd27b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c1066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:60170/0 (socket says 172.21.15.194:60170) 2024-01-27T23:31:47.512 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd27b7fe700 1 -- 172.21.15.194:0/1462924826 learned_addr learned my addr 172.21.15.194:0/1462924826 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:47.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd27b7fe700 1 -- 172.21.15.194:0/1462924826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd264005040 con 0x7fd27c060280 2024-01-27T23:31:47.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd27b7fe700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c1066e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd264009830 tx=0x7fd26400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:47.513 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd273fff700 1 -- 172.21.15.194:0/1462924826 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd264009320 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd273fff700 1 -- 172.21.15.194:0/1462924826 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd2640136c0 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd27c106dd0 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd273fff700 1 -- 172.21.15.194:0/1462924826 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd26401ee90 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.519+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd27c107260 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.520+0000 7fd273fff700 1 -- 172.21.15.194:0/1462924826 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd26401e420 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.520+0000 7fd273fff700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd26802ff10 0x7fd2680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.520+0000 7fd273fff700 1 -- 172.21.15.194:0/1462924826 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd264040fa0 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.521+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd27c059100 con 0x7fd27c060280 2024-01-27T23:31:47.514 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.521+0000 7fd27affd700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd26802ff10 0x7fd2680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:47.515 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.521+0000 7fd27affd700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd26802ff10 0x7fd2680323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fd26c0096a0 tx=0x7fd26c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:47.518 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.525+0000 7fd273fff700 1 -- 172.21.15.194:0/1462924826 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd26401c930 con 0x7fd27c060280 2024-01-27T23:31:47.723 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.729+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd27c045ad0 con 0x7fd27c060280 2024-01-27T23:31:47.723 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.730+0000 7fd273fff700 1 -- 172.21.15.194:0/1462924826 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd26401cb40 con 0x7fd27c060280 2024-01-27T23:31:47.724 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:47.724 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd26802ff10 msgr2=0x7fd2680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd26802ff10 0x7fd2680323d0 secure :-1 s=READY pgs=140 cs=0 l=1 rev1=1 rx=0x7fd26c0096a0 tx=0x7fd26c006b40).stop 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 msgr2=0x7fd27c1066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c1066e0 secure :-1 s=READY pgs=348 cs=0 l=1 rev1=1 rx=0x7fd264009830 tx=0x7fd26400b040).stop 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 shutdown_connections 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd26802ff10 0x7fd2680323d0 unknown :-1 s=CLOSED pgs=140 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 --2- 172.21.15.194:0/1462924826 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd27c060280 0x7fd27c1066e0 unknown :-1 s=CLOSED pgs=348 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:47.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.731+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 >> 172.21.15.194:0/1462924826 conn(0x7fd27c0f0b70 msgr2=0x7fd27c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:47.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.732+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 shutdown_connections 2024-01-27T23:31:47.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:47.732+0000 7fd282117700 1 -- 172.21.15.194:0/1462924826 wait complete. 2024-01-27T23:31:47.726 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:48.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:47.317846+0000 mgr.smithi121.hcppvz (mgr.14162) 201 : cluster [DBG] pgmap v189: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:48.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:47.731378+0000 mon.smithi121 (mon.0) 268 : audit [DBG] from='client.? 172.21.15.194:0/1462924826' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:49.108 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:49.108 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 -- 172.21.15.194:0/2116395672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 msgr2=0x7f87d00f4ac0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 --2- 172.21.15.194:0/2116395672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d00f4ac0 secure :-1 s=READY pgs=349 cs=0 l=1 rev1=1 rx=0x7f87c0009fb0 tx=0x7f87c0009b30).stop 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 -- 172.21.15.194:0/2116395672 shutdown_connections 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 --2- 172.21.15.194:0/2116395672 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d00f4ac0 unknown :-1 s=CLOSED pgs=349 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 -- 172.21.15.194:0/2116395672 >> 172.21.15.194:0/2116395672 conn(0x7f87d00f0120 msgr2=0x7f87d00f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 -- 172.21.15.194:0/2116395672 shutdown_connections 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 -- 172.21.15.194:0/2116395672 wait complete. 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 Processor -- start 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.299+0000 7f87cf59e700 1 -- start start 2024-01-27T23:31:50.294 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87cf59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:50.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87cf59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f87d0106bd0 con 0x7f87d00f46a0 2024-01-27T23:31:50.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87ce59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:50.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87ce59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:60174/0 (socket says 172.21.15.194:60174) 2024-01-27T23:31:50.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87ce59c700 1 -- 172.21.15.194:0/3960410190 learned_addr learned my addr 172.21.15.194:0/3960410190 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:50.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87ce59c700 1 -- 172.21.15.194:0/3960410190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f87c0005040 con 0x7f87d00f46a0 2024-01-27T23:31:50.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87ce59c700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d01066e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f87c0013510 tx=0x7f87c000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:50.295 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87beffd700 1 -- 172.21.15.194:0/3960410190 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87c0009320 con 0x7f87d00f46a0 2024-01-27T23:31:50.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f87d0106dd0 con 0x7f87d00f46a0 2024-01-27T23:31:50.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.300+0000 7f87beffd700 1 -- 172.21.15.194:0/3960410190 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f87c0013f00 con 0x7f87d00f46a0 2024-01-27T23:31:50.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.301+0000 7f87beffd700 1 -- 172.21.15.194:0/3960410190 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f87c0014dc0 con 0x7f87d00f46a0 2024-01-27T23:31:50.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.301+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f87d0107260 con 0x7f87d00f46a0 2024-01-27T23:31:50.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.301+0000 7f87beffd700 1 -- 172.21.15.194:0/3960410190 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f87c00095f0 con 0x7f87d00f46a0 2024-01-27T23:31:50.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.301+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f87d0045ad0 con 0x7f87d00f46a0 2024-01-27T23:31:50.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.301+0000 7f87beffd700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f87b802ff60 0x7f87b8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:50.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.301+0000 7f87beffd700 1 -- 172.21.15.194:0/3960410190 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f87c0041000 con 0x7f87d00f46a0 2024-01-27T23:31:50.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.302+0000 7f87cdd9b700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f87b802ff60 0x7f87b8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:50.297 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.302+0000 7f87cdd9b700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f87b802ff60 0x7f87b8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f87c40096a0 tx=0x7f87c4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:50.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.306+0000 7f87beffd700 1 -- 172.21.15.194:0/3960410190 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f87c0014420 con 0x7f87d00f46a0 2024-01-27T23:31:50.502 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.509+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f87d005eb50 con 0x7f87d00f46a0 2024-01-27T23:31:50.503 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.510+0000 7f87beffd700 1 -- 172.21.15.194:0/3960410190 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f87c0010230 con 0x7f87d00f46a0 2024-01-27T23:31:50.503 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:50.503 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f87b802ff60 msgr2=0x7f87b8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f87b802ff60 0x7f87b8032420 secure :-1 s=READY pgs=141 cs=0 l=1 rev1=1 rx=0x7f87c40096a0 tx=0x7f87c4006b40).stop 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 msgr2=0x7f87d01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d01066e0 secure :-1 s=READY pgs=350 cs=0 l=1 rev1=1 rx=0x7f87c0013510 tx=0x7f87c000b040).stop 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 shutdown_connections 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f87b802ff60 0x7f87b8032420 unknown :-1 s=CLOSED pgs=141 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 --2- 172.21.15.194:0/3960410190 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f87d00f46a0 0x7f87d01066e0 unknown :-1 s=CLOSED pgs=350 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:50.505 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 >> 172.21.15.194:0/3960410190 conn(0x7f87d00f0120 msgr2=0x7f87d00f0c00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:50.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 shutdown_connections 2024-01-27T23:31:50.506 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:50.511+0000 7f87cf59e700 1 -- 172.21.15.194:0/3960410190 wait complete. 2024-01-27T23:31:50.506 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:50.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:49.318110+0000 mgr.smithi121.hcppvz (mgr.14162) 202 : cluster [DBG] pgmap v190: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:51.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:51 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:50.510999+0000 mon.smithi121 (mon.0) 269 : audit [DBG] from='client.? 172.21.15.194:0/3960410190' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:51.918 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:51.918 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:52.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:51.318410+0000 mgr.smithi121.hcppvz (mgr.14162) 203 : cluster [DBG] pgmap v191: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.116+0000 7f6e08a78700 1 -- 172.21.15.194:0/1372396371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 msgr2=0x7f6e040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.116+0000 7f6e08a78700 1 --2- 172.21.15.194:0/1372396371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e040f5830 secure :-1 s=READY pgs=351 cs=0 l=1 rev1=1 rx=0x7f6dec009fb0 tx=0x7f6dec009b30).stop 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.116+0000 7f6e08a78700 1 -- 172.21.15.194:0/1372396371 shutdown_connections 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.116+0000 7f6e08a78700 1 --2- 172.21.15.194:0/1372396371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e040f5830 unknown :-1 s=CLOSED pgs=351 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.116+0000 7f6e08a78700 1 -- 172.21.15.194:0/1372396371 >> 172.21.15.194:0/1372396371 conn(0x7f6e040f0b70 msgr2=0x7f6e040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.116+0000 7f6e08a78700 1 -- 172.21.15.194:0/1372396371 shutdown_connections 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.116+0000 7f6e08a78700 1 -- 172.21.15.194:0/1372396371 wait complete. 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e08a78700 1 Processor -- start 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e08a78700 1 -- start start 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e08a78700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e04106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:53.112 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e08a78700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6e04106c10 con 0x7f6e040f5410 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e04106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e0259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e04106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:60198/0 (socket says 172.21.15.194:60198) 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e0259c700 1 -- 172.21.15.194:0/3291439896 learned_addr learned my addr 172.21.15.194:0/3291439896 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.117+0000 7f6e0259c700 1 -- 172.21.15.194:0/3291439896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6dec005040 con 0x7f6e040f5410 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.118+0000 7f6e0259c700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e04106720 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f6dec013510 tx=0x7f6dec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.118+0000 7f6dfaffd700 1 -- 172.21.15.194:0/3291439896 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dec009320 con 0x7f6e040f5410 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.118+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6e04106e10 con 0x7f6e040f5410 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.118+0000 7f6dfaffd700 1 -- 172.21.15.194:0/3291439896 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6dec013f00 con 0x7f6e040f5410 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.118+0000 7f6dfaffd700 1 -- 172.21.15.194:0/3291439896 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6dec014d90 con 0x7f6e040f5410 2024-01-27T23:31:53.113 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.118+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6e041072a0 con 0x7f6e040f5410 2024-01-27T23:31:53.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.119+0000 7f6dfaffd700 1 -- 172.21.15.194:0/3291439896 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6dec0095f0 con 0x7f6e040f5410 2024-01-27T23:31:53.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.119+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6e04045ad0 con 0x7f6e040f5410 2024-01-27T23:31:53.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.119+0000 7f6dfaffd700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6df002ff60 0x7f6df0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:53.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.119+0000 7f6dfaffd700 1 -- 172.21.15.194:0/3291439896 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6dec041000 con 0x7f6e040f5410 2024-01-27T23:31:53.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.119+0000 7f6e01d9b700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6df002ff60 0x7f6df0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:53.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.119+0000 7f6e01d9b700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6df002ff60 0x7f6df0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6df40096a0 tx=0x7f6df4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:53.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.124+0000 7f6dfaffd700 1 -- 172.21.15.194:0/3291439896 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6dec01d7f0 con 0x7f6e040f5410 2024-01-27T23:31:53.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.331+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6e04108150 con 0x7f6e040f5410 2024-01-27T23:31:53.325 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.332+0000 7f6dfaffd700 1 -- 172.21.15.194:0/3291439896 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6dec010230 con 0x7f6e040f5410 2024-01-27T23:31:53.325 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:53.326 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6df002ff60 msgr2=0x7f6df0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6df002ff60 0x7f6df0032420 secure :-1 s=READY pgs=142 cs=0 l=1 rev1=1 rx=0x7f6df40096a0 tx=0x7f6df4006b40).stop 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 msgr2=0x7f6e04106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e04106720 secure :-1 s=READY pgs=352 cs=0 l=1 rev1=1 rx=0x7f6dec013510 tx=0x7f6dec00b040).stop 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 shutdown_connections 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6df002ff60 0x7f6df0032420 unknown :-1 s=CLOSED pgs=142 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 --2- 172.21.15.194:0/3291439896 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6e040f5410 0x7f6e04106720 unknown :-1 s=CLOSED pgs=352 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 >> 172.21.15.194:0/3291439896 conn(0x7f6e040f0b70 msgr2=0x7f6e0405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:53.327 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 shutdown_connections 2024-01-27T23:31:53.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:53.333+0000 7f6e08a78700 1 -- 172.21.15.194:0/3291439896 wait complete. 2024-01-27T23:31:53.328 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:53.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:53 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:53.332998+0000 mon.smithi121 (mon.0) 270 : audit [DBG] from='client.? 172.21.15.194:0/3291439896' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:54.716 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:54.716 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:54.840 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:53.318764+0000 mgr.smithi121.hcppvz (mgr.14162) 204 : cluster [DBG] pgmap v192: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:55.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 -- 172.21.15.194:0/2126975310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 msgr2=0x7f66500f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:55.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 --2- 172.21.15.194:0/2126975310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500f55c0 secure :-1 s=READY pgs=353 cs=0 l=1 rev1=1 rx=0x7f6640009fb0 tx=0x7f6640009b30).stop 2024-01-27T23:31:55.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 -- 172.21.15.194:0/2126975310 shutdown_connections 2024-01-27T23:31:55.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 --2- 172.21.15.194:0/2126975310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500f55c0 unknown :-1 s=CLOSED pgs=353 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:55.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 -- 172.21.15.194:0/2126975310 >> 172.21.15.194:0/2126975310 conn(0x7f66500f09a0 msgr2=0x7f66500f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:55.857 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 -- 172.21.15.194:0/2126975310 shutdown_connections 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 -- 172.21.15.194:0/2126975310 wait complete. 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 Processor -- start 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 -- start start 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500fd120 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.862+0000 7f664f59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f66500fb5a0 con 0x7f66500f51a0 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f664e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500fd120 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f664e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500fd120 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53402/0 (socket says 172.21.15.194:53402) 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f664e59c700 1 -- 172.21.15.194:0/1937502773 learned_addr learned my addr 172.21.15.194:0/1937502773 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f664e59c700 1 -- 172.21.15.194:0/1937502773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6640005040 con 0x7f66500f51a0 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f664e59c700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500fd120 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6640013510 tx=0x7f664000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f663effd700 1 -- 172.21.15.194:0/1937502773 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6640009320 con 0x7f66500f51a0 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f663effd700 1 -- 172.21.15.194:0/1937502773 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6640013f00 con 0x7f66500f51a0 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f663effd700 1 -- 172.21.15.194:0/1937502773 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6640014dc0 con 0x7f66500f51a0 2024-01-27T23:31:55.858 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f66500fb740 con 0x7f66500f51a0 2024-01-27T23:31:55.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.863+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f66500fbbd0 con 0x7f66500f51a0 2024-01-27T23:31:55.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.864+0000 7f663effd700 1 -- 172.21.15.194:0/1937502773 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f66400095f0 con 0x7f66500f51a0 2024-01-27T23:31:55.859 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.864+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6650059100 con 0x7f66500f51a0 2024-01-27T23:31:55.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.864+0000 7f663effd700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f663802ff60 0x7f6638032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:55.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.864+0000 7f663effd700 1 -- 172.21.15.194:0/1937502773 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6640041100 con 0x7f66500f51a0 2024-01-27T23:31:55.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.864+0000 7f664dd9b700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f663802ff60 0x7f6638032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:55.860 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.865+0000 7f664dd9b700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f663802ff60 0x7f6638032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f66440096a0 tx=0x7f6644006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:55.862 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:55.869+0000 7f663effd700 1 -- 172.21.15.194:0/1937502773 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6640014420 con 0x7f66500f51a0 2024-01-27T23:31:56.068 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.075+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f66500fabe0 con 0x7f66500f51a0 2024-01-27T23:31:56.069 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.075+0000 7f663effd700 1 -- 172.21.15.194:0/1937502773 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6640010230 con 0x7f66500f51a0 2024-01-27T23:31:56.069 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:56.069 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:56.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f663802ff60 msgr2=0x7f6638032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:56.070 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f663802ff60 0x7f6638032420 secure :-1 s=READY pgs=143 cs=0 l=1 rev1=1 rx=0x7f66440096a0 tx=0x7f6644006b40).stop 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 msgr2=0x7f66500fd120 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500fd120 secure :-1 s=READY pgs=354 cs=0 l=1 rev1=1 rx=0x7f6640013510 tx=0x7f664000b040).stop 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 shutdown_connections 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f663802ff60 0x7f6638032420 unknown :-1 s=CLOSED pgs=143 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 --2- 172.21.15.194:0/1937502773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f66500f51a0 0x7f66500fd120 unknown :-1 s=CLOSED pgs=354 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 >> 172.21.15.194:0/1937502773 conn(0x7f66500f09a0 msgr2=0x7f665005fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 shutdown_connections 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:56.077+0000 7f664f59e700 1 -- 172.21.15.194:0/1937502773 wait complete. 2024-01-27T23:31:56.071 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:56.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:55.319124+0000 mgr.smithi121.hcppvz (mgr.14162) 205 : cluster [DBG] pgmap v193: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:56.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:56.076482+0000 mon.smithi121 (mon.0) 271 : audit [DBG] from='client.? 172.21.15.194:0/1937502773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:31:57.452 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:31:57.452 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.563+0000 7ff8a9282700 1 -- 172.21.15.194:0/3680117751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 msgr2=0x7ff8a40f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.563+0000 7ff8a9282700 1 --2- 172.21.15.194:0/3680117751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a40f55c0 secure :-1 s=READY pgs=355 cs=0 l=1 rev1=1 rx=0x7ff894009fb0 tx=0x7ff894009b30).stop 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.563+0000 7ff8a9282700 1 -- 172.21.15.194:0/3680117751 shutdown_connections 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.563+0000 7ff8a9282700 1 --2- 172.21.15.194:0/3680117751 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a40f55c0 unknown :-1 s=CLOSED pgs=355 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.563+0000 7ff8a9282700 1 -- 172.21.15.194:0/3680117751 >> 172.21.15.194:0/3680117751 conn(0x7ff8a40f09a0 msgr2=0x7ff8a40f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.563+0000 7ff8a9282700 1 -- 172.21.15.194:0/3680117751 shutdown_connections 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.564+0000 7ff8a9282700 1 -- 172.21.15.194:0/3680117751 wait complete. 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.564+0000 7ff8a9282700 1 Processor -- start 2024-01-27T23:31:58.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.564+0000 7ff8a9282700 1 -- start start 2024-01-27T23:31:58.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.564+0000 7ff8a9282700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a41064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:58.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.564+0000 7ff8a9282700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff8a41069d0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.560 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a41064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a41064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53422/0 (socket says 172.21.15.194:53422) 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a3fff700 1 -- 172.21.15.194:0/3491671476 learned_addr learned my addr 172.21.15.194:0/3491671476 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a3fff700 1 -- 172.21.15.194:0/3491671476 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff894005040 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a3fff700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a41064e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ff894013510 tx=0x7ff89400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a0ff9700 1 -- 172.21.15.194:0/3491671476 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff894009320 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff8a4106bd0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a0ff9700 1 -- 172.21.15.194:0/3491671476 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff894013f00 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a0ff9700 1 -- 172.21.15.194:0/3491671476 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff894014dc0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.565+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff8a4107060 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.566+0000 7ff8a0ff9700 1 -- 172.21.15.194:0/3491671476 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff89401d3d0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.566+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff8a4045ad0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.566+0000 7ff8a0ff9700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff88c02ff60 0x7ff88c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:31:58.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.566+0000 7ff8a0ff9700 1 -- 172.21.15.194:0/3491671476 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff8940413f0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.567+0000 7ff8a37fe700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff88c02ff60 0x7ff88c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:31:58.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.567+0000 7ff8a37fe700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff88c02ff60 0x7ff88c032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff8980096a0 tx=0x7ff898006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:31:58.564 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.571+0000 7ff8a0ff9700 1 -- 172.21.15.194:0/3491671476 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff8940126f0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.778+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff8a4107eb0 con 0x7ff8a40f51a0 2024-01-27T23:31:58.772 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.779+0000 7ff8a0ff9700 1 -- 172.21.15.194:0/3491671476 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff894012330 con 0x7ff8a40f51a0 2024-01-27T23:31:58.773 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:31:58.773 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:31:58.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff88c02ff60 msgr2=0x7ff88c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:58.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff88c02ff60 0x7ff88c032420 secure :-1 s=READY pgs=144 cs=0 l=1 rev1=1 rx=0x7ff8980096a0 tx=0x7ff898006b40).stop 2024-01-27T23:31:58.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 msgr2=0x7ff8a41064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:31:58.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a41064e0 secure :-1 s=READY pgs=356 cs=0 l=1 rev1=1 rx=0x7ff894013510 tx=0x7ff89400b040).stop 2024-01-27T23:31:58.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 shutdown_connections 2024-01-27T23:31:58.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff88c02ff60 0x7ff88c032420 unknown :-1 s=CLOSED pgs=144 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:58.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 --2- 172.21.15.194:0/3491671476 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff8a40f51a0 0x7ff8a41064e0 unknown :-1 s=CLOSED pgs=356 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:31:58.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 >> 172.21.15.194:0/3491671476 conn(0x7ff8a40f09a0 msgr2=0x7ff8a405fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:31:58.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 shutdown_connections 2024-01-27T23:31:58.775 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:31:58.780+0000 7ff8a9282700 1 -- 172.21.15.194:0/3491671476 wait complete. 2024-01-27T23:31:58.775 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:31:58.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:57.319487+0000 mgr.smithi121.hcppvz (mgr.14162) 206 : cluster [DBG] pgmap v194: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:31:59.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:31:59 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:31:58.780010+0000 mon.smithi121 (mon.0) 272 : audit [DBG] from='client.? 172.21.15.194:0/3491671476' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:00.151 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:00.151 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:00.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:31:59.319714+0000 mgr.smithi121.hcppvz (mgr.14162) 207 : cluster [DBG] pgmap v195: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:01.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.319+0000 7f305a561700 1 -- 172.21.15.194:0/2768056580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 msgr2=0x7f30540f9150 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:01.314 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.319+0000 7f305a561700 1 --2- 172.21.15.194:0/2768056580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30540f9150 secure :-1 s=READY pgs=357 cs=0 l=1 rev1=1 rx=0x7f3044009fb0 tx=0x7f3044009b30).stop 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.319+0000 7f305a561700 1 -- 172.21.15.194:0/2768056580 shutdown_connections 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.319+0000 7f305a561700 1 --2- 172.21.15.194:0/2768056580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30540f9150 unknown :-1 s=CLOSED pgs=357 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.319+0000 7f305a561700 1 -- 172.21.15.194:0/2768056580 >> 172.21.15.194:0/2768056580 conn(0x7f30540f0b70 msgr2=0x7f30540f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.319+0000 7f305a561700 1 -- 172.21.15.194:0/2768056580 shutdown_connections 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.320+0000 7f305a561700 1 -- 172.21.15.194:0/2768056580 wait complete. 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.320+0000 7f305a561700 1 Processor -- start 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.320+0000 7f305a561700 1 -- start start 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.320+0000 7f305a561700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30541066b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.320+0000 7f305a561700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f3054106ba0 con 0x7f3054060220 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3053fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30541066b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3053fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30541066b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53432/0 (socket says 172.21.15.194:53432) 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3053fff700 1 -- 172.21.15.194:0/2438829805 learned_addr learned my addr 172.21.15.194:0/2438829805 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:01.315 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3053fff700 1 -- 172.21.15.194:0/2438829805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f3044005040 con 0x7f3054060220 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3053fff700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30541066b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3044013510 tx=0x7f304400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3050ff9700 1 -- 172.21.15.194:0/2438829805 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3044009320 con 0x7f3054060220 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f3054106da0 con 0x7f3054060220 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3050ff9700 1 -- 172.21.15.194:0/2438829805 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f3044013f00 con 0x7f3054060220 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f3050ff9700 1 -- 172.21.15.194:0/2438829805 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f3044014dc0 con 0x7f3054060220 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.321+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f3054107230 con 0x7f3054060220 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.322+0000 7f3050ff9700 1 -- 172.21.15.194:0/2438829805 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f304401d3d0 con 0x7f3054060220 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.322+0000 7f3050ff9700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f303c02ffb0 0x7f303c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:01.317 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.322+0000 7f3050ff9700 1 -- 172.21.15.194:0/2438829805 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f30440410f0 con 0x7f3054060220 2024-01-27T23:32:01.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.322+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f3054045ad0 con 0x7f3054060220 2024-01-27T23:32:01.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.322+0000 7f30537fe700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f303c02ffb0 0x7f303c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:01.318 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.323+0000 7f30537fe700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f303c02ffb0 0x7f303c032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f30480096a0 tx=0x7f3048006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:01.320 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.327+0000 7f3050ff9700 1 -- 172.21.15.194:0/2438829805 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f3044012900 con 0x7f3054060220 2024-01-27T23:32:01.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.532+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f3054107f50 con 0x7f3054060220 2024-01-27T23:32:01.526 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.533+0000 7f3050ff9700 1 -- 172.21.15.194:0/2438829805 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f304402b020 con 0x7f3054060220 2024-01-27T23:32:01.526 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:01.527 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:01.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f303c02ffb0 msgr2=0x7f303c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:01.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f303c02ffb0 0x7f303c032470 secure :-1 s=READY pgs=145 cs=0 l=1 rev1=1 rx=0x7f30480096a0 tx=0x7f3048006b40).stop 2024-01-27T23:32:01.528 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 msgr2=0x7f30541066b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30541066b0 secure :-1 s=READY pgs=358 cs=0 l=1 rev1=1 rx=0x7f3044013510 tx=0x7f304400b040).stop 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 shutdown_connections 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f303c02ffb0 0x7f303c032470 unknown :-1 s=CLOSED pgs=145 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 --2- 172.21.15.194:0/2438829805 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f3054060220 0x7f30541066b0 unknown :-1 s=CLOSED pgs=358 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 >> 172.21.15.194:0/2438829805 conn(0x7f30540f0b70 msgr2=0x7f30540f1650 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 shutdown_connections 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:01.534+0000 7f305a561700 1 -- 172.21.15.194:0/2438829805 wait complete. 2024-01-27T23:32:01.529 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:02.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:01.320000+0000 mgr.smithi121.hcppvz (mgr.14162) 208 : cluster [DBG] pgmap v196: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:02.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:01.534068+0000 mon.smithi121 (mon.0) 273 : audit [DBG] from='client.? 172.21.15.194:0/2438829805' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:02.922 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:02.922 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.119+0000 7f8a0c016700 1 -- 172.21.15.194:0/119186259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 msgr2=0x7f8a040f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.119+0000 7f8a0c016700 1 --2- 172.21.15.194:0/119186259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a040f5830 secure :-1 s=READY pgs=359 cs=0 l=1 rev1=1 rx=0x7f89f4009fb0 tx=0x7f89f4009b30).stop 2024-01-27T23:32:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.119+0000 7f8a0c016700 1 -- 172.21.15.194:0/119186259 shutdown_connections 2024-01-27T23:32:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.119+0000 7f8a0c016700 1 --2- 172.21.15.194:0/119186259 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a040f5830 unknown :-1 s=CLOSED pgs=359 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.119+0000 7f8a0c016700 1 -- 172.21.15.194:0/119186259 >> 172.21.15.194:0/119186259 conn(0x7f8a040f0b70 msgr2=0x7f8a040f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.119+0000 7f8a0c016700 1 -- 172.21.15.194:0/119186259 shutdown_connections 2024-01-27T23:32:04.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.119+0000 7f8a0c016700 1 -- 172.21.15.194:0/119186259 wait complete. 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.120+0000 7f8a0c016700 1 Processor -- start 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.120+0000 7f8a0c016700 1 -- start start 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.120+0000 7f8a0c016700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a04105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.120+0000 7f8a0c016700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8a04105af0 con 0x7f8a040f5410 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.120+0000 7f8a09db2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a04105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.120+0000 7f8a09db2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a04105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:53440/0 (socket says 172.21.15.194:53440) 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.120+0000 7f8a09db2700 1 -- 172.21.15.194:0/116911524 learned_addr learned my addr 172.21.15.194:0/116911524 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.121+0000 7f8a09db2700 1 -- 172.21.15.194:0/116911524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f89f4005040 con 0x7f8a040f5410 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.121+0000 7f8a09db2700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a04105600 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f89f4013510 tx=0x7f89f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.121+0000 7f89fa7fc700 1 -- 172.21.15.194:0/116911524 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f4009320 con 0x7f8a040f5410 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.121+0000 7f89fa7fc700 1 -- 172.21.15.194:0/116911524 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f89f4013f00 con 0x7f8a040f5410 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.121+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8a04105cf0 con 0x7f8a040f5410 2024-01-27T23:32:04.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.121+0000 7f89fa7fc700 1 -- 172.21.15.194:0/116911524 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f89f4014d90 con 0x7f8a040f5410 2024-01-27T23:32:04.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.121+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8a04106180 con 0x7f8a040f5410 2024-01-27T23:32:04.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.122+0000 7f89fa7fc700 1 -- 172.21.15.194:0/116911524 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f89f4014420 con 0x7f8a040f5410 2024-01-27T23:32:04.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.122+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8a04045ad0 con 0x7f8a040f5410 2024-01-27T23:32:04.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.122+0000 7f89fa7fc700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f89f002ff60 0x7f89f0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:04.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.122+0000 7f89fa7fc700 1 -- 172.21.15.194:0/116911524 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f89f40410b0 con 0x7f8a040f5410 2024-01-27T23:32:04.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.122+0000 7f8a095b1700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f89f002ff60 0x7f89f0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:04.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.123+0000 7f8a095b1700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f89f002ff60 0x7f89f0032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8a000096a0 tx=0x7f8a00006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:04.120 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.127+0000 7f89fa7fc700 1 -- 172.21.15.194:0/116911524 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f89f40095f0 con 0x7f8a040f5410 2024-01-27T23:32:04.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.332+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8a040fae60 con 0x7f8a040f5410 2024-01-27T23:32:04.326 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.332+0000 7f89fa7fc700 1 -- 172.21.15.194:0/116911524 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f89f401db70 con 0x7f8a040f5410 2024-01-27T23:32:04.327 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:04.327 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f89f002ff60 msgr2=0x7f89f0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f89f002ff60 0x7f89f0032420 secure :-1 s=READY pgs=146 cs=0 l=1 rev1=1 rx=0x7f8a000096a0 tx=0x7f8a00006b40).stop 2024-01-27T23:32:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 msgr2=0x7f8a04105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a04105600 secure :-1 s=READY pgs=360 cs=0 l=1 rev1=1 rx=0x7f89f4013510 tx=0x7f89f400b040).stop 2024-01-27T23:32:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 shutdown_connections 2024-01-27T23:32:04.328 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f89f002ff60 0x7f89f0032420 unknown :-1 s=CLOSED pgs=146 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 --2- 172.21.15.194:0/116911524 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f8a040f5410 0x7f8a04105600 unknown :-1 s=CLOSED pgs=360 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 >> 172.21.15.194:0/116911524 conn(0x7f8a040f0b70 msgr2=0x7f8a040f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 shutdown_connections 2024-01-27T23:32:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:04.334+0000 7f8a0c016700 1 -- 172.21.15.194:0/116911524 wait complete. 2024-01-27T23:32:04.329 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:04.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:03.320245+0000 mgr.smithi121.hcppvz (mgr.14162) 209 : cluster [DBG] pgmap v197: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:04.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:04.333818+0000 mon.smithi121 (mon.0) 274 : audit [DBG] from='client.? 172.21.15.194:0/116911524' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:05.712 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:05.712 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:06.800 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.806+0000 7fed6b1d4700 1 -- 172.21.15.194:0/866561482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 msgr2=0x7fed640f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.806+0000 7fed6b1d4700 1 --2- 172.21.15.194:0/866561482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed640f5830 secure :-1 s=READY pgs=361 cs=0 l=1 rev1=1 rx=0x7fed58009fb0 tx=0x7fed58009b30).stop 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.806+0000 7fed6b1d4700 1 -- 172.21.15.194:0/866561482 shutdown_connections 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.806+0000 7fed6b1d4700 1 --2- 172.21.15.194:0/866561482 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed640f5830 unknown :-1 s=CLOSED pgs=361 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.806+0000 7fed6b1d4700 1 -- 172.21.15.194:0/866561482 >> 172.21.15.194:0/866561482 conn(0x7fed640f0b70 msgr2=0x7fed640f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.806+0000 7fed6b1d4700 1 -- 172.21.15.194:0/866561482 shutdown_connections 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.806+0000 7fed6b1d4700 1 -- 172.21.15.194:0/866561482 wait complete. 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.807+0000 7fed6b1d4700 1 Processor -- start 2024-01-27T23:32:06.801 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.807+0000 7fed6b1d4700 1 -- start start 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.807+0000 7fed6b1d4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed64105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.807+0000 7fed6b1d4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fed64105b20 con 0x7fed640f5410 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.807+0000 7fed68f70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed64105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.807+0000 7fed68f70700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed64105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:43192/0 (socket says 172.21.15.194:43192) 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.807+0000 7fed68f70700 1 -- 172.21.15.194:0/1066777425 learned_addr learned my addr 172.21.15.194:0/1066777425 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.808+0000 7fed68f70700 1 -- 172.21.15.194:0/1066777425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fed58005040 con 0x7fed640f5410 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.808+0000 7fed68f70700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed64105630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fed58013510 tx=0x7fed5800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:06.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.808+0000 7fed617fa700 1 -- 172.21.15.194:0/1066777425 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed58009320 con 0x7fed640f5410 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.808+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fed64105d20 con 0x7fed640f5410 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.808+0000 7fed617fa700 1 -- 172.21.15.194:0/1066777425 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fed58013f00 con 0x7fed640f5410 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.808+0000 7fed617fa700 1 -- 172.21.15.194:0/1066777425 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fed58014d90 con 0x7fed640f5410 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.808+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fed641061b0 con 0x7fed640f5410 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.809+0000 7fed617fa700 1 -- 172.21.15.194:0/1066777425 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fed58014420 con 0x7fed640f5410 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.809+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fed64045ad0 con 0x7fed640f5410 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.809+0000 7fed617fa700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fed5402ffb0 0x7fed54032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:06.804 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.809+0000 7fed617fa700 1 -- 172.21.15.194:0/1066777425 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fed58040fd0 con 0x7fed640f5410 2024-01-27T23:32:06.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.809+0000 7fed63fff700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fed5402ffb0 0x7fed54032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:06.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.810+0000 7fed63fff700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fed5402ffb0 0x7fed54032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fed500096a0 tx=0x7fed50006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:06.807 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:06.814+0000 7fed617fa700 1 -- 172.21.15.194:0/1066777425 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fed5806d2b0 con 0x7fed640f5410 2024-01-27T23:32:06.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:05.320448+0000 mgr.smithi121.hcppvz (mgr.14162) 210 : cluster [DBG] pgmap v198: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:07.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.019+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fed640fae60 con 0x7fed640f5410 2024-01-27T23:32:07.014 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.020+0000 7fed617fa700 1 -- 172.21.15.194:0/1066777425 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fed580095f0 con 0x7fed640f5410 2024-01-27T23:32:07.014 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:07.014 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:07.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fed5402ffb0 msgr2=0x7fed54032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:07.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fed5402ffb0 0x7fed54032470 secure :-1 s=READY pgs=147 cs=0 l=1 rev1=1 rx=0x7fed500096a0 tx=0x7fed50006b40).stop 2024-01-27T23:32:07.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 msgr2=0x7fed64105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:07.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed64105630 secure :-1 s=READY pgs=362 cs=0 l=1 rev1=1 rx=0x7fed58013510 tx=0x7fed5800b040).stop 2024-01-27T23:32:07.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 shutdown_connections 2024-01-27T23:32:07.016 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fed5402ffb0 0x7fed54032470 unknown :-1 s=CLOSED pgs=147 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:07.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 --2- 172.21.15.194:0/1066777425 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fed640f5410 0x7fed64105630 unknown :-1 s=CLOSED pgs=362 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:07.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 >> 172.21.15.194:0/1066777425 conn(0x7fed640f0b70 msgr2=0x7fed640f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:07.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 shutdown_connections 2024-01-27T23:32:07.017 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:07.022+0000 7fed6b1d4700 1 -- 172.21.15.194:0/1066777425 wait complete. 2024-01-27T23:32:07.017 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:07.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:07 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:07.021514+0000 mon.smithi121 (mon.0) 275 : audit [DBG] from='client.? 172.21.15.194:0/1066777425' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:08.406 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:08.406 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:08.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:07.320733+0000 mgr.smithi121.hcppvz (mgr.14162) 211 : cluster [DBG] pgmap v199: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:09.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.594+0000 7fb80f326700 1 -- 172.21.15.194:0/3999144292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 msgr2=0x7fb8080f6ea0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.594+0000 7fb80f326700 1 --2- 172.21.15.194:0/3999144292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb8080f6ea0 secure :-1 s=READY pgs=363 cs=0 l=1 rev1=1 rx=0x7fb7f8009fb0 tx=0x7fb7f8009b30).stop 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.594+0000 7fb80f326700 1 -- 172.21.15.194:0/3999144292 shutdown_connections 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.594+0000 7fb80f326700 1 --2- 172.21.15.194:0/3999144292 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb8080f6ea0 unknown :-1 s=CLOSED pgs=363 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.594+0000 7fb80f326700 1 -- 172.21.15.194:0/3999144292 >> 172.21.15.194:0/3999144292 conn(0x7fb8080f02e0 msgr2=0x7fb8080f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.594+0000 7fb80f326700 1 -- 172.21.15.194:0/3999144292 shutdown_connections 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.594+0000 7fb80f326700 1 -- 172.21.15.194:0/3999144292 wait complete. 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80f326700 1 Processor -- start 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80f326700 1 -- start start 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80f326700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb808109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80f326700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8080f5310 con 0x7fb8080f6a80 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80d0c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb808109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:09.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80d0c2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb808109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:43208/0 (socket says 172.21.15.194:43208) 2024-01-27T23:32:09.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80d0c2700 1 -- 172.21.15.194:0/1465901405 learned_addr learned my addr 172.21.15.194:0/1465901405 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:09.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.595+0000 7fb80d0c2700 1 -- 172.21.15.194:0/1465901405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb7f8005040 con 0x7fb8080f6a80 2024-01-27T23:32:09.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.596+0000 7fb80d0c2700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb808109430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb7f8013510 tx=0x7fb7f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:09.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.596+0000 7fb7fdffb700 1 -- 172.21.15.194:0/1465901405 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f8009320 con 0x7fb8080f6a80 2024-01-27T23:32:09.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.596+0000 7fb7fdffb700 1 -- 172.21.15.194:0/1465901405 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb7f8013f00 con 0x7fb8080f6a80 2024-01-27T23:32:09.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.596+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8080f5510 con 0x7fb8080f6a80 2024-01-27T23:32:09.591 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.596+0000 7fb7fdffb700 1 -- 172.21.15.194:0/1465901405 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb7f8014dc0 con 0x7fb8080f6a80 2024-01-27T23:32:09.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.596+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb808109ae0 con 0x7fb8080f6a80 2024-01-27T23:32:09.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.597+0000 7fb7fdffb700 1 -- 172.21.15.194:0/1465901405 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb7f80095f0 con 0x7fb8080f6a80 2024-01-27T23:32:09.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.597+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb808059100 con 0x7fb8080f6a80 2024-01-27T23:32:09.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.597+0000 7fb7fdffb700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:09.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.597+0000 7fb7fdffb700 1 -- 172.21.15.194:0/1465901405 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb7f8041040 con 0x7fb8080f6a80 2024-01-27T23:32:09.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.597+0000 7fb80c8c1700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:09.592 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.597+0000 7fb80c8c1700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb7f402ff60 0x7fb7f4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb8040096a0 tx=0x7fb804006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:09.595 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.601+0000 7fb7fdffb700 1 -- 172.21.15.194:0/1465901405 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb7f8014420 con 0x7fb8080f6a80 2024-01-27T23:32:09.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.809+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb808045ad0 con 0x7fb8080f6a80 2024-01-27T23:32:09.803 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.810+0000 7fb7fdffb700 1 -- 172.21.15.194:0/1465901405 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb7f8010230 con 0x7fb8080f6a80 2024-01-27T23:32:09.803 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:09.804 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb7f402ff60 msgr2=0x7fb7f4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb7f402ff60 0x7fb7f4032420 secure :-1 s=READY pgs=148 cs=0 l=1 rev1=1 rx=0x7fb8040096a0 tx=0x7fb804006b40).stop 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 msgr2=0x7fb808109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb808109430 secure :-1 s=READY pgs=364 cs=0 l=1 rev1=1 rx=0x7fb7f8013510 tx=0x7fb7f800b040).stop 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 shutdown_connections 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb7f402ff60 0x7fb7f4032420 unknown :-1 s=CLOSED pgs=148 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 --2- 172.21.15.194:0/1465901405 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8080f6a80 0x7fb808109430 unknown :-1 s=CLOSED pgs=364 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:09.805 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 >> 172.21.15.194:0/1465901405 conn(0x7fb8080f02e0 msgr2=0x7fb8080f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:09.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 shutdown_connections 2024-01-27T23:32:09.806 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:09.811+0000 7fb80f326700 1 -- 172.21.15.194:0/1465901405 wait complete. 2024-01-27T23:32:09.806 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:10.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:09.321039+0000 mgr.smithi121.hcppvz (mgr.14162) 212 : cluster [DBG] pgmap v200: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:10.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:09.810897+0000 mon.smithi121 (mon.0) 276 : audit [DBG] from='client.? 172.21.15.194:0/1465901405' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:11.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:11.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:12.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.401+0000 7fb09b214700 1 -- 172.21.15.194:0/2044582323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 msgr2=0x7fb0940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:12.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.401+0000 7fb09b214700 1 --2- 172.21.15.194:0/2044582323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb0940f6930 secure :-1 s=READY pgs=365 cs=0 l=1 rev1=1 rx=0x7fb08c009960 tx=0x7fb08c004d10).stop 2024-01-27T23:32:12.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.401+0000 7fb09b214700 1 -- 172.21.15.194:0/2044582323 shutdown_connections 2024-01-27T23:32:12.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.401+0000 7fb09b214700 1 --2- 172.21.15.194:0/2044582323 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb0940f6930 unknown :-1 s=CLOSED pgs=365 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:12.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.401+0000 7fb09b214700 1 -- 172.21.15.194:0/2044582323 >> 172.21.15.194:0/2044582323 conn(0x7fb0940f1c70 msgr2=0x7fb0940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:12.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.401+0000 7fb09b214700 1 -- 172.21.15.194:0/2044582323 shutdown_connections 2024-01-27T23:32:12.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.401+0000 7fb09b214700 1 -- 172.21.15.194:0/2044582323 wait complete. 2024-01-27T23:32:12.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.402+0000 7fb09b214700 1 Processor -- start 2024-01-27T23:32:12.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.402+0000 7fb09b214700 1 -- start start 2024-01-27T23:32:12.397 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.402+0000 7fb09b214700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.402+0000 7fb09b214700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb094106c20 con 0x7fb0940f6510 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.402+0000 7fb098fb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb098fb0700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:43226/0 (socket says 172.21.15.194:43226) 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb098fb0700 1 -- 172.21.15.194:0/3463946854 learned_addr learned my addr 172.21.15.194:0/3463946854 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb098fb0700 1 -- 172.21.15.194:0/3463946854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb08c009640 con 0x7fb0940f6510 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb098fb0700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb094106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fb08c004b70 tx=0x7fb08c004d10).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb0917fa700 1 -- 172.21.15.194:0/3463946854 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb08c018070 con 0x7fb0940f6510 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb0917fa700 1 -- 172.21.15.194:0/3463946854 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb08c00b870 con 0x7fb0940f6510 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb094106e20 con 0x7fb0940f6510 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb0917fa700 1 -- 172.21.15.194:0/3463946854 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb08c01c560 con 0x7fb0940f6510 2024-01-27T23:32:12.398 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.403+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb0941072b0 con 0x7fb0940f6510 2024-01-27T23:32:12.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.404+0000 7fb0917fa700 1 -- 172.21.15.194:0/3463946854 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb08c013630 con 0x7fb0940f6510 2024-01-27T23:32:12.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.404+0000 7fb0917fa700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb07c02ff60 0x7fb07c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:12.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.404+0000 7fb0917fa700 1 -- 172.21.15.194:0/3463946854 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb08c0411d0 con 0x7fb0940f6510 2024-01-27T23:32:12.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.404+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb094045ad0 con 0x7fb0940f6510 2024-01-27T23:32:12.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.404+0000 7fb093fff700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb07c02ff60 0x7fb07c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:12.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.405+0000 7fb093fff700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb07c02ff60 0x7fb07c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fb0880096a0 tx=0x7fb088006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:12.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.409+0000 7fb0917fa700 1 -- 172.21.15.194:0/3463946854 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb08c004810 con 0x7fb0940f6510 2024-01-27T23:32:12.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:11.321407+0000 mgr.smithi121.hcppvz (mgr.14162) 213 : cluster [DBG] pgmap v201: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:12.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.612+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb0940fc050 con 0x7fb0940f6510 2024-01-27T23:32:12.606 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.613+0000 7fb0917fa700 1 -- 172.21.15.194:0/3463946854 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb08c012380 con 0x7fb0940f6510 2024-01-27T23:32:12.607 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:12.607 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:12.608 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb07c02ff60 msgr2=0x7fb07c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb07c02ff60 0x7fb07c032420 secure :-1 s=READY pgs=149 cs=0 l=1 rev1=1 rx=0x7fb0880096a0 tx=0x7fb088006b40).stop 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 msgr2=0x7fb094106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb094106730 secure :-1 s=READY pgs=366 cs=0 l=1 rev1=1 rx=0x7fb08c004b70 tx=0x7fb08c004d10).stop 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 shutdown_connections 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb07c02ff60 0x7fb07c032420 unknown :-1 s=CLOSED pgs=149 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 --2- 172.21.15.194:0/3463946854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb0940f6510 0x7fb094106730 unknown :-1 s=CLOSED pgs=366 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 >> 172.21.15.194:0/3463946854 conn(0x7fb0940f1c70 msgr2=0x7fb0940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 shutdown_connections 2024-01-27T23:32:12.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:12.614+0000 7fb09b214700 1 -- 172.21.15.194:0/3463946854 wait complete. 2024-01-27T23:32:12.610 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:13.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:13 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:12.614026+0000 mon.smithi121 (mon.0) 277 : audit [DBG] from='client.? 172.21.15.194:0/3463946854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:13.999 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:13.999 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:14.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:13.321771+0000 mgr.smithi121.hcppvz (mgr.14162) 214 : cluster [DBG] pgmap v202: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:15.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.178+0000 7f4d7dbca700 1 -- 172.21.15.194:0/2603391759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 msgr2=0x7f4d780f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:15.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.178+0000 7f4d7dbca700 1 --2- 172.21.15.194:0/2603391759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d780f55c0 secure :-1 s=READY pgs=367 cs=0 l=1 rev1=1 rx=0x7f4d6c009fb0 tx=0x7f4d6c009b30).stop 2024-01-27T23:32:15.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.178+0000 7f4d7dbca700 1 -- 172.21.15.194:0/2603391759 shutdown_connections 2024-01-27T23:32:15.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.178+0000 7f4d7dbca700 1 --2- 172.21.15.194:0/2603391759 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d780f55c0 unknown :-1 s=CLOSED pgs=367 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:15.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.178+0000 7f4d7dbca700 1 -- 172.21.15.194:0/2603391759 >> 172.21.15.194:0/2603391759 conn(0x7f4d780f09a0 msgr2=0x7f4d780f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:15.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.178+0000 7f4d7dbca700 1 -- 172.21.15.194:0/2603391759 shutdown_connections 2024-01-27T23:32:15.173 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.178+0000 7f4d7dbca700 1 -- 172.21.15.194:0/2603391759 wait complete. 2024-01-27T23:32:15.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.179+0000 7f4d7dbca700 1 Processor -- start 2024-01-27T23:32:15.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.179+0000 7f4d7dbca700 1 -- start start 2024-01-27T23:32:15.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.179+0000 7f4d7dbca700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d781064a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:15.174 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.179+0000 7f4d7dbca700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4d78106990 con 0x7f4d780f51a0 2024-01-27T23:32:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.179+0000 7f4d7cbc8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d781064a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d7cbc8700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d781064a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:43238/0 (socket says 172.21.15.194:43238) 2024-01-27T23:32:15.175 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d7cbc8700 1 -- 172.21.15.194:0/698914371 learned_addr learned my addr 172.21.15.194:0/698914371 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d7cbc8700 1 -- 172.21.15.194:0/698914371 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4d6c005040 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d7cbc8700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d781064a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4d6c009830 tx=0x7f4d6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d757fa700 1 -- 172.21.15.194:0/698914371 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c009320 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d757fa700 1 -- 172.21.15.194:0/698914371 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4d6c0136c0 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4d78106b90 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d757fa700 1 -- 172.21.15.194:0/698914371 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4d6c01ee90 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.180+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4d78107020 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.181+0000 7f4d757fa700 1 -- 172.21.15.194:0/698914371 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4d6c014440 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.181+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4d78045ad0 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.181+0000 7f4d757fa700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d6802ff10 0x7f4d680323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.181+0000 7f4d757fa700 1 -- 172.21.15.194:0/698914371 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4d6c040ec0 con 0x7f4d780f51a0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.181+0000 7f4d77fff700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d6802ff10 0x7f4d680323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:15.176 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.182+0000 7f4d77fff700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d6802ff10 0x7f4d680323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:15.179 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.186+0000 7f4d757fa700 1 -- 172.21.15.194:0/698914371 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4d6c01c910 con 0x7f4d780f51a0 2024-01-27T23:32:15.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.393+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4d78107ed0 con 0x7f4d780f51a0 2024-01-27T23:32:15.387 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.393+0000 7f4d757fa700 1 -- 172.21.15.194:0/698914371 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4d6c02c2d0 con 0x7f4d780f51a0 2024-01-27T23:32:15.387 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:15.388 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:15.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d6802ff10 msgr2=0x7f4d680323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:15.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d6802ff10 0x7f4d680323d0 secure :-1 s=READY pgs=150 cs=0 l=1 rev1=1 rx=0x7f4d640096a0 tx=0x7f4d64006b40).stop 2024-01-27T23:32:15.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 msgr2=0x7f4d781064a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:15.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d781064a0 secure :-1 s=READY pgs=368 cs=0 l=1 rev1=1 rx=0x7f4d6c009830 tx=0x7f4d6c00b040).stop 2024-01-27T23:32:15.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 shutdown_connections 2024-01-27T23:32:15.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4d6802ff10 0x7f4d680323d0 unknown :-1 s=CLOSED pgs=150 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:15.389 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 --2- 172.21.15.194:0/698914371 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4d780f51a0 0x7f4d781064a0 unknown :-1 s=CLOSED pgs=368 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:15.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 >> 172.21.15.194:0/698914371 conn(0x7f4d780f09a0 msgr2=0x7f4d7805fde0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:15.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 shutdown_connections 2024-01-27T23:32:15.390 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:15.395+0000 7f4d7dbca700 1 -- 172.21.15.194:0/698914371 wait complete. 2024-01-27T23:32:15.390 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:16.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:16.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:16.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:15.322094+0000 mgr.smithi121.hcppvz (mgr.14162) 215 : cluster [DBG] pgmap v203: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:16.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:15.394739+0000 mon.smithi121 (mon.0) 278 : audit [DBG] from='client.? 172.21.15.194:0/698914371' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.916+0000 7f199adf9700 1 -- 172.21.15.194:0/3556938227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 msgr2=0x7f19940f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.916+0000 7f199adf9700 1 --2- 172.21.15.194:0/3556938227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f19940f6930 secure :-1 s=READY pgs=369 cs=0 l=1 rev1=1 rx=0x7f1988009fb0 tx=0x7f1988009b30).stop 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.916+0000 7f199adf9700 1 -- 172.21.15.194:0/3556938227 shutdown_connections 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.916+0000 7f199adf9700 1 --2- 172.21.15.194:0/3556938227 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f19940f6930 unknown :-1 s=CLOSED pgs=369 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.916+0000 7f199adf9700 1 -- 172.21.15.194:0/3556938227 >> 172.21.15.194:0/3556938227 conn(0x7f19940f1c70 msgr2=0x7f19940f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.916+0000 7f199adf9700 1 -- 172.21.15.194:0/3556938227 shutdown_connections 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.916+0000 7f199adf9700 1 -- 172.21.15.194:0/3556938227 wait complete. 2024-01-27T23:32:17.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.917+0000 7f199adf9700 1 Processor -- start 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.917+0000 7f199adf9700 1 -- start start 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.917+0000 7f199adf9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f1994106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f199adf9700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1994106c20 con 0x7f19940f6510 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f1998b95700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f1994106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f1998b95700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f1994106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:50736/0 (socket says 172.21.15.194:50736) 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f1998b95700 1 -- 172.21.15.194:0/2394784850 learned_addr learned my addr 172.21.15.194:0/2394784850 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f1998b95700 1 -- 172.21.15.194:0/2394784850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1988005040 con 0x7f19940f6510 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f1998b95700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f1994106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1988013510 tx=0x7f198800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f19917fa700 1 -- 172.21.15.194:0/2394784850 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1988009320 con 0x7f19940f6510 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1994106e20 con 0x7f19940f6510 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f19917fa700 1 -- 172.21.15.194:0/2394784850 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1988013f00 con 0x7f19940f6510 2024-01-27T23:32:17.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f19917fa700 1 -- 172.21.15.194:0/2394784850 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1988014d90 con 0x7f19940f6510 2024-01-27T23:32:17.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.918+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f19941072b0 con 0x7f19940f6510 2024-01-27T23:32:17.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.919+0000 7f19917fa700 1 -- 172.21.15.194:0/2394784850 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f19880095f0 con 0x7f19940f6510 2024-01-27T23:32:17.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.919+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1994045ad0 con 0x7f19940f6510 2024-01-27T23:32:17.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.919+0000 7f19917fa700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f198402ff60 0x7f1984032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:17.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.919+0000 7f19917fa700 1 -- 172.21.15.194:0/2394784850 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1988041080 con 0x7f19940f6510 2024-01-27T23:32:17.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.920+0000 7f1993fff700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f198402ff60 0x7f1984032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:17.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.920+0000 7f1993fff700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f198402ff60 0x7f1984032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f19800096a0 tx=0x7f1980006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:17.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:17.924+0000 7f19917fa700 1 -- 172.21.15.194:0/2394784850 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1988014420 con 0x7f19940f6510 2024-01-27T23:32:18.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.132+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f19940fbf60 con 0x7f19940f6510 2024-01-27T23:32:18.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.133+0000 7f19917fa700 1 -- 172.21.15.194:0/2394784850 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f198801d7f0 con 0x7f19940f6510 2024-01-27T23:32:18.127 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:18.127 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:18.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f198402ff60 msgr2=0x7f1984032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:18.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f198402ff60 0x7f1984032420 secure :-1 s=READY pgs=151 cs=0 l=1 rev1=1 rx=0x7f19800096a0 tx=0x7f1980006b40).stop 2024-01-27T23:32:18.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 msgr2=0x7f1994106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f1994106730 secure :-1 s=READY pgs=370 cs=0 l=1 rev1=1 rx=0x7f1988013510 tx=0x7f198800b040).stop 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 shutdown_connections 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f198402ff60 0x7f1984032420 unknown :-1 s=CLOSED pgs=151 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 --2- 172.21.15.194:0/2394784850 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f19940f6510 0x7f1994106730 unknown :-1 s=CLOSED pgs=370 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 >> 172.21.15.194:0/2394784850 conn(0x7f19940f1c70 msgr2=0x7f19940f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 shutdown_connections 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:18.134+0000 7f199adf9700 1 -- 172.21.15.194:0/2394784850 wait complete. 2024-01-27T23:32:18.129 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:18.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:17.322432+0000 mgr.smithi121.hcppvz (mgr.14162) 216 : cluster [DBG] pgmap v204: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:18.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:17.398560+0000 mon.smithi121 (mon.0) 279 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:32:18.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:17.406355+0000 mon.smithi121 (mon.0) 280 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:32:18.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:18.134070+0000 mon.smithi121 (mon.0) 281 : audit [DBG] from='client.? 172.21.15.194:0/2394784850' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:19.510 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:19.510 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.707+0000 7f75e6927700 1 -- 172.21.15.194:0/1488155959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 msgr2=0x7f75e00f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.707+0000 7f75e6927700 1 --2- 172.21.15.194:0/1488155959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e00f55c0 secure :-1 s=READY pgs=371 cs=0 l=1 rev1=1 rx=0x7f75dc009fb0 tx=0x7f75dc009b30).stop 2024-01-27T23:32:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 -- 172.21.15.194:0/1488155959 shutdown_connections 2024-01-27T23:32:20.703 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 --2- 172.21.15.194:0/1488155959 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e00f55c0 unknown :-1 s=CLOSED pgs=371 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 -- 172.21.15.194:0/1488155959 >> 172.21.15.194:0/1488155959 conn(0x7f75e00f09a0 msgr2=0x7f75e00f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 -- 172.21.15.194:0/1488155959 shutdown_connections 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 -- 172.21.15.194:0/1488155959 wait complete. 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 Processor -- start 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 -- start start 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e01086d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.708+0000 7f75e6927700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f75e0108bc0 con 0x7f75e00f51a0 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75e5925700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e01086d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75e5925700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e01086d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:50746/0 (socket says 172.21.15.194:50746) 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75e5925700 1 -- 172.21.15.194:0/2407432977 learned_addr learned my addr 172.21.15.194:0/2407432977 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75e5925700 1 -- 172.21.15.194:0/2407432977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f75dc005040 con 0x7f75e00f51a0 2024-01-27T23:32:20.704 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75e5925700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e01086d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f75dc013510 tx=0x7f75dc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75d67fc700 1 -- 172.21.15.194:0/2407432977 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75dc009320 con 0x7f75e00f51a0 2024-01-27T23:32:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f75e0108dc0 con 0x7f75e00f51a0 2024-01-27T23:32:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75d67fc700 1 -- 172.21.15.194:0/2407432977 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f75dc013f00 con 0x7f75e00f51a0 2024-01-27T23:32:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75d67fc700 1 -- 172.21.15.194:0/2407432977 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f75dc014dc0 con 0x7f75e00f51a0 2024-01-27T23:32:20.705 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.709+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75e0109250 con 0x7f75e00f51a0 2024-01-27T23:32:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.710+0000 7f75d67fc700 1 -- 172.21.15.194:0/2407432977 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f75dc0095f0 con 0x7f75e00f51a0 2024-01-27T23:32:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.710+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f75e0045ad0 con 0x7f75e00f51a0 2024-01-27T23:32:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.710+0000 7f75d67fc700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75cc02ff60 0x7f75cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.710+0000 7f75d67fc700 1 -- 172.21.15.194:0/2407432977 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f75dc0410e0 con 0x7f75e00f51a0 2024-01-27T23:32:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.710+0000 7f75e5124700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75cc02ff60 0x7f75cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:20.706 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.711+0000 7f75e5124700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75cc02ff60 0x7f75cc032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f75d00096a0 tx=0x7f75d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:20.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.715+0000 7f75d67fc700 1 -- 172.21.15.194:0/2407432977 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f75dc014420 con 0x7f75e00f51a0 2024-01-27T23:32:20.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:19.322721+0000 mgr.smithi121.hcppvz (mgr.14162) 217 : cluster [DBG] pgmap v205: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:20.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.920+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75e010a100 con 0x7f75e00f51a0 2024-01-27T23:32:20.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.921+0000 7f75d67fc700 1 -- 172.21.15.194:0/2407432977 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f75dc01c920 con 0x7f75e00f51a0 2024-01-27T23:32:20.915 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:20.915 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.922+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75cc02ff60 msgr2=0x7f75cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.922+0000 7f75e6927700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75cc02ff60 0x7f75cc032420 secure :-1 s=READY pgs=152 cs=0 l=1 rev1=1 rx=0x7f75d00096a0 tx=0x7f75d0006b40).stop 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.922+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 msgr2=0x7f75e01086d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.922+0000 7f75e6927700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e01086d0 secure :-1 s=READY pgs=372 cs=0 l=1 rev1=1 rx=0x7f75dc013510 tx=0x7f75dc00b040).stop 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.923+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 shutdown_connections 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.923+0000 7f75e6927700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f75cc02ff60 0x7f75cc032420 unknown :-1 s=CLOSED pgs=152 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.923+0000 7f75e6927700 1 --2- 172.21.15.194:0/2407432977 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75e00f51a0 0x7f75e01086d0 unknown :-1 s=CLOSED pgs=372 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.923+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 >> 172.21.15.194:0/2407432977 conn(0x7f75e00f09a0 msgr2=0x7f75e005fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.923+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 shutdown_connections 2024-01-27T23:32:20.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:20.923+0000 7f75e6927700 1 -- 172.21.15.194:0/2407432977 wait complete. 2024-01-27T23:32:20.918 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:21.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:21 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:20.922440+0000 mon.smithi121 (mon.0) 282 : audit [DBG] from='client.? 172.21.15.194:0/2407432977' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:22.338 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:22.338 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:22.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:21.323064+0000 mgr.smithi121.hcppvz (mgr.14162) 218 : cluster [DBG] pgmap v206: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.563+0000 7f76e0808700 1 -- 172.21.15.194:0/1393080214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 msgr2=0x7f76dc0f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.563+0000 7f76e0808700 1 --2- 172.21.15.194:0/1393080214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc0f66c0 secure :-1 s=READY pgs=373 cs=0 l=1 rev1=1 rx=0x7f76cc009fb0 tx=0x7f76cc009b30).stop 2024-01-27T23:32:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.563+0000 7f76e0808700 1 -- 172.21.15.194:0/1393080214 shutdown_connections 2024-01-27T23:32:23.558 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.563+0000 7f76e0808700 1 --2- 172.21.15.194:0/1393080214 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc0f66c0 unknown :-1 s=CLOSED pgs=373 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.563+0000 7f76e0808700 1 -- 172.21.15.194:0/1393080214 >> 172.21.15.194:0/1393080214 conn(0x7f76dc0f1aa0 msgr2=0x7f76dc0f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.563+0000 7f76e0808700 1 -- 172.21.15.194:0/1393080214 shutdown_connections 2024-01-27T23:32:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.563+0000 7f76e0808700 1 -- 172.21.15.194:0/1393080214 wait complete. 2024-01-27T23:32:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.564+0000 7f76e0808700 1 Processor -- start 2024-01-27T23:32:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.564+0000 7f76e0808700 1 -- start start 2024-01-27T23:32:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.564+0000 7f76e0808700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:23.559 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.564+0000 7f76e0808700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f76dc069e20 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76db7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:50760/0 (socket says 172.21.15.194:50760) 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76db7fe700 1 -- 172.21.15.194:0/3599568870 learned_addr learned my addr 172.21.15.194:0/3599568870 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76db7fe700 1 -- 172.21.15.194:0/3599568870 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76cc005040 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76db7fe700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc06b9a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f76cc009830 tx=0x7f76cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76c3fff700 1 -- 172.21.15.194:0/3599568870 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76cc009320 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f76dc069fc0 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76c3fff700 1 -- 172.21.15.194:0/3599568870 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76cc0136c0 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76c3fff700 1 -- 172.21.15.194:0/3599568870 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76cc01ee90 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.565+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f76dc06a450 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.566+0000 7f76c3fff700 1 -- 172.21.15.194:0/3599568870 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76cc01e420 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.566+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f76dc059100 con 0x7f76dc0f62a0 2024-01-27T23:32:23.561 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.566+0000 7f76c3fff700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76c402ff10 0x7f76c40323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:23.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.566+0000 7f76c3fff700 1 -- 172.21.15.194:0/3599568870 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76cc041050 con 0x7f76dc0f62a0 2024-01-27T23:32:23.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.566+0000 7f76daffd700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76c402ff10 0x7f76c40323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:23.562 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.567+0000 7f76daffd700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76c402ff10 0x7f76c40323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f76d00096a0 tx=0x7f76d0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:23.565 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.571+0000 7f76c3fff700 1 -- 172.21.15.194:0/3599568870 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76cc06d2b0 con 0x7f76dc0f62a0 2024-01-27T23:32:23.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.777+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f76dc0fbce0 con 0x7f76dc0f62a0 2024-01-27T23:32:23.771 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.778+0000 7f76c3fff700 1 -- 172.21.15.194:0/3599568870 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76cc0095f0 con 0x7f76dc0f62a0 2024-01-27T23:32:23.772 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:23.772 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:23.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76c402ff10 msgr2=0x7f76c40323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:23.773 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76c402ff10 0x7f76c40323d0 secure :-1 s=READY pgs=153 cs=0 l=1 rev1=1 rx=0x7f76d00096a0 tx=0x7f76d0006b40).stop 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 msgr2=0x7f76dc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc06b9a0 secure :-1 s=READY pgs=374 cs=0 l=1 rev1=1 rx=0x7f76cc009830 tx=0x7f76cc00b040).stop 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 shutdown_connections 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76c402ff10 0x7f76c40323d0 unknown :-1 s=CLOSED pgs=153 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 --2- 172.21.15.194:0/3599568870 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f76dc0f62a0 0x7f76dc06b9a0 unknown :-1 s=CLOSED pgs=374 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 >> 172.21.15.194:0/3599568870 conn(0x7f76dc0f1aa0 msgr2=0x7f76dc0f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 shutdown_connections 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:23.779+0000 7f76e0808700 1 -- 172.21.15.194:0/3599568870 wait complete. 2024-01-27T23:32:23.774 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:24.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:23.323354+0000 mgr.smithi121.hcppvz (mgr.14162) 219 : cluster [DBG] pgmap v207: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:24.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:23.778929+0000 mon.smithi121 (mon.0) 283 : audit [DBG] from='client.? 172.21.15.194:0/3599568870' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:25.206 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:25.207 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:26.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 -- 172.21.15.194:0/2737902932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 msgr2=0x7f67100f4960 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 --2- 172.21.15.194:0/2737902932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f67100f4960 secure :-1 s=READY pgs=375 cs=0 l=1 rev1=1 rx=0x7f6704009fb0 tx=0x7f6704009b30).stop 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 -- 172.21.15.194:0/2737902932 shutdown_connections 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 --2- 172.21.15.194:0/2737902932 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f67100f4960 unknown :-1 s=CLOSED pgs=375 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 -- 172.21.15.194:0/2737902932 >> 172.21.15.194:0/2737902932 conn(0x7f67100effc0 msgr2=0x7f67100f23e0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 -- 172.21.15.194:0/2737902932 shutdown_connections 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 -- 172.21.15.194:0/2737902932 wait complete. 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 Processor -- start 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.375+0000 7f670f59e700 1 -- start start 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670f59e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f6710106480 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670f59e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6710106970 con 0x7f67100f4540 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f6710106480 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670e59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f6710106480 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:59136/0 (socket says 172.21.15.194:59136) 2024-01-27T23:32:26.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670e59c700 1 -- 172.21.15.194:0/1730378291 learned_addr learned my addr 172.21.15.194:0/1730378291 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670e59c700 1 -- 172.21.15.194:0/1730378291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6704005040 con 0x7f67100f4540 2024-01-27T23:32:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670e59c700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f6710106480 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6704014000 tx=0x7f670400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f66feffd700 1 -- 172.21.15.194:0/1730378291 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f67040095f0 con 0x7f67100f4540 2024-01-27T23:32:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6710106b70 con 0x7f67100f4540 2024-01-27T23:32:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f66feffd700 1 -- 172.21.15.194:0/1730378291 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6704007e70 con 0x7f67100f4540 2024-01-27T23:32:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f66feffd700 1 -- 172.21.15.194:0/1730378291 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6704014ea0 con 0x7f67100f4540 2024-01-27T23:32:26.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.376+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6710107000 con 0x7f67100f4540 2024-01-27T23:32:26.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.377+0000 7f66feffd700 1 -- 172.21.15.194:0/1730378291 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f670401d4e0 con 0x7f67100f4540 2024-01-27T23:32:26.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.377+0000 7f66feffd700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66f802ff60 0x7f66f8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:26.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.377+0000 7f66feffd700 1 -- 172.21.15.194:0/1730378291 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6704040900 con 0x7f67100f4540 2024-01-27T23:32:26.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.377+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6710045ad0 con 0x7f67100f4540 2024-01-27T23:32:26.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.378+0000 7f670dd9b700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66f802ff60 0x7f66f8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:26.373 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.378+0000 7f670dd9b700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66f802ff60 0x7f66f8032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f67000096a0 tx=0x7f6700006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:26.376 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.382+0000 7f66feffd700 1 -- 172.21.15.194:0/1730378291 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f670400f8f0 con 0x7f67100f4540 2024-01-27T23:32:26.586 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.592+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6710107dc0 con 0x7f67100f4540 2024-01-27T23:32:26.587 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.593+0000 7f66feffd700 1 -- 172.21.15.194:0/1730378291 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6704012330 con 0x7f67100f4540 2024-01-27T23:32:26.587 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:26.587 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.594+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66f802ff60 msgr2=0x7f66f8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.594+0000 7f670f59e700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66f802ff60 0x7f66f8032420 secure :-1 s=READY pgs=154 cs=0 l=1 rev1=1 rx=0x7f67000096a0 tx=0x7f6700006b40).stop 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.594+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 msgr2=0x7f6710106480 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.594+0000 7f670f59e700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f6710106480 secure :-1 s=READY pgs=376 cs=0 l=1 rev1=1 rx=0x7f6704014000 tx=0x7f670400b040).stop 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.594+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 shutdown_connections 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.595+0000 7f670f59e700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f66f802ff60 0x7f66f8032420 unknown :-1 s=CLOSED pgs=154 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.595+0000 7f670f59e700 1 --2- 172.21.15.194:0/1730378291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f67100f4540 0x7f6710106480 unknown :-1 s=CLOSED pgs=376 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:26.589 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.595+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 >> 172.21.15.194:0/1730378291 conn(0x7f67100effc0 msgr2=0x7f67100f0ac0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:26.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.595+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 shutdown_connections 2024-01-27T23:32:26.590 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:26.595+0000 7f670f59e700 1 -- 172.21.15.194:0/1730378291 wait complete. 2024-01-27T23:32:26.590 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:26.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:25.323699+0000 mgr.smithi121.hcppvz (mgr.14162) 220 : cluster [DBG] pgmap v208: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:27.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:27 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:26.594328+0000 mon.smithi121 (mon.0) 284 : audit [DBG] from='client.? 172.21.15.194:0/1730378291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:27.975 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:27.976 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:28.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:27.324042+0000 mgr.smithi121.hcppvz (mgr.14162) 221 : cluster [DBG] pgmap v209: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:29.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.156+0000 7f29c0aee700 1 -- 172.21.15.194:0/882161858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 msgr2=0x7f29bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:29.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.156+0000 7f29c0aee700 1 --2- 172.21.15.194:0/882161858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc0f6930 secure :-1 s=READY pgs=377 cs=0 l=1 rev1=1 rx=0x7f29a4009fb0 tx=0x7f29a4009b30).stop 2024-01-27T23:32:29.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.156+0000 7f29c0aee700 1 -- 172.21.15.194:0/882161858 shutdown_connections 2024-01-27T23:32:29.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.156+0000 7f29c0aee700 1 --2- 172.21.15.194:0/882161858 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc0f6930 unknown :-1 s=CLOSED pgs=377 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:29.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.156+0000 7f29c0aee700 1 -- 172.21.15.194:0/882161858 >> 172.21.15.194:0/882161858 conn(0x7f29bc0f1c70 msgr2=0x7f29bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:29.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.156+0000 7f29c0aee700 1 -- 172.21.15.194:0/882161858 shutdown_connections 2024-01-27T23:32:29.151 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.156+0000 7f29c0aee700 1 -- 172.21.15.194:0/882161858 wait complete. 2024-01-27T23:32:29.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.157+0000 7f29c0aee700 1 Processor -- start 2024-01-27T23:32:29.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.157+0000 7f29c0aee700 1 -- start start 2024-01-27T23:32:29.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.157+0000 7f29c0aee700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:29.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.157+0000 7f29c0aee700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f29bc069e20 con 0x7f29bc0f6510 2024-01-27T23:32:29.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.157+0000 7f29ba59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:29.153 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.157+0000 7f29ba59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:59162/0 (socket says 172.21.15.194:59162) 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.157+0000 7f29ba59c700 1 -- 172.21.15.194:0/925669624 learned_addr learned my addr 172.21.15.194:0/925669624 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.158+0000 7f29ba59c700 1 -- 172.21.15.194:0/925669624 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f29a4005040 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.158+0000 7f29ba59c700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f29a4014000 tx=0x7f29a400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.158+0000 7f29b2ffd700 1 -- 172.21.15.194:0/925669624 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29a40095f0 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.158+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f29bc069fc0 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.158+0000 7f29b2ffd700 1 -- 172.21.15.194:0/925669624 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f29a4007e70 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.158+0000 7f29b2ffd700 1 -- 172.21.15.194:0/925669624 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f29a4014ea0 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.158+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f29bc06a450 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.159+0000 7f29b2ffd700 1 -- 172.21.15.194:0/925669624 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f29a4025070 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.159+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f29bc059100 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.159+0000 7f29b2ffd700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f29a802ff60 0x7f29a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.159+0000 7f29b2ffd700 1 -- 172.21.15.194:0/925669624 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f29a4040a30 con 0x7f29bc0f6510 2024-01-27T23:32:29.154 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.159+0000 7f29b9d9b700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f29a802ff60 0x7f29a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:29.155 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.160+0000 7f29b9d9b700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f29a802ff60 0x7f29a8032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f29ac0096a0 tx=0x7f29ac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:29.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.164+0000 7f29b2ffd700 1 -- 172.21.15.194:0/925669624 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f29a40079c0 con 0x7f29bc0f6510 2024-01-27T23:32:29.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.370+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f29bc0fbf60 con 0x7f29bc0f6510 2024-01-27T23:32:29.365 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.371+0000 7f29b2ffd700 1 -- 172.21.15.194:0/925669624 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f29a400f300 con 0x7f29bc0f6510 2024-01-27T23:32:29.365 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:29.365 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f29a802ff60 msgr2=0x7f29a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f29a802ff60 0x7f29a8032420 secure :-1 s=READY pgs=155 cs=0 l=1 rev1=1 rx=0x7f29ac0096a0 tx=0x7f29ac006b40).stop 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 msgr2=0x7f29bc06b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 secure :-1 s=READY pgs=378 cs=0 l=1 rev1=1 rx=0x7f29a4014000 tx=0x7f29a400b040).stop 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 shutdown_connections 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f29a802ff60 0x7f29a8032420 unknown :-1 s=CLOSED pgs=155 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 --2- 172.21.15.194:0/925669624 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f29bc0f6510 0x7f29bc06b9a0 unknown :-1 s=CLOSED pgs=378 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.372+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 >> 172.21.15.194:0/925669624 conn(0x7f29bc0f1c70 msgr2=0x7f29bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.373+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 shutdown_connections 2024-01-27T23:32:29.367 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:29.373+0000 7f29c0aee700 1 -- 172.21.15.194:0/925669624 wait complete. 2024-01-27T23:32:29.368 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:29.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:29 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:29.371975+0000 mon.smithi121 (mon.0) 285 : audit [DBG] from='client.? 172.21.15.194:0/925669624' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:30.749 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:30.749 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:30.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:29.324476+0000 mgr.smithi121.hcppvz (mgr.14162) 222 : cluster [DBG] pgmap v210: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:31.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.881+0000 7f60c47cd700 1 -- 172.21.15.194:0/735675695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 msgr2=0x7f60bc0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:31.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.881+0000 7f60c47cd700 1 --2- 172.21.15.194:0/735675695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc0f6930 secure :-1 s=READY pgs=379 cs=0 l=1 rev1=1 rx=0x7f60ac009fb0 tx=0x7f60ac009b30).stop 2024-01-27T23:32:31.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.881+0000 7f60c47cd700 1 -- 172.21.15.194:0/735675695 shutdown_connections 2024-01-27T23:32:31.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.881+0000 7f60c47cd700 1 --2- 172.21.15.194:0/735675695 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc0f6930 unknown :-1 s=CLOSED pgs=379 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:31.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.881+0000 7f60c47cd700 1 -- 172.21.15.194:0/735675695 >> 172.21.15.194:0/735675695 conn(0x7f60bc0f1c70 msgr2=0x7f60bc0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:31.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.881+0000 7f60c47cd700 1 -- 172.21.15.194:0/735675695 shutdown_connections 2024-01-27T23:32:31.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.881+0000 7f60c47cd700 1 -- 172.21.15.194:0/735675695 wait complete. 2024-01-27T23:32:31.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.882+0000 7f60c47cd700 1 Processor -- start 2024-01-27T23:32:31.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.882+0000 7f60c47cd700 1 -- start start 2024-01-27T23:32:31.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.882+0000 7f60c47cd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:31.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.882+0000 7f60c47cd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f60bc106c20 con 0x7f60bc0f6510 2024-01-27T23:32:31.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.882+0000 7f60c2569700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.882+0000 7f60c2569700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:59184/0 (socket says 172.21.15.194:59184) 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.882+0000 7f60c2569700 1 -- 172.21.15.194:0/265558600 learned_addr learned my addr 172.21.15.194:0/265558600 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.883+0000 7f60c2569700 1 -- 172.21.15.194:0/265558600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f60ac005040 con 0x7f60bc0f6510 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.883+0000 7f60c2569700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f60ac013510 tx=0x7f60ac00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.883+0000 7f60b2ffd700 1 -- 172.21.15.194:0/265558600 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac009320 con 0x7f60bc0f6510 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.883+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f60bc106e20 con 0x7f60bc0f6510 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.883+0000 7f60b2ffd700 1 -- 172.21.15.194:0/265558600 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f60ac013f00 con 0x7f60bc0f6510 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.883+0000 7f60b2ffd700 1 -- 172.21.15.194:0/265558600 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f60ac014d90 con 0x7f60bc0f6510 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.883+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f60bc1072b0 con 0x7f60bc0f6510 2024-01-27T23:32:31.879 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.884+0000 7f60b2ffd700 1 -- 172.21.15.194:0/265558600 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f60ac0095f0 con 0x7f60bc0f6510 2024-01-27T23:32:31.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.884+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f60bc045ad0 con 0x7f60bc0f6510 2024-01-27T23:32:31.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.884+0000 7f60b2ffd700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f60a802ff60 0x7f60a8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:31.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.884+0000 7f60b2ffd700 1 -- 172.21.15.194:0/265558600 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f60ac041000 con 0x7f60bc0f6510 2024-01-27T23:32:31.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.884+0000 7f60c1d68700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f60a802ff60 0x7f60a8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:31.880 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.885+0000 7f60c1d68700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f60a802ff60 0x7f60a8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f60b80096a0 tx=0x7f60b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:31.882 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:31.889+0000 7f60b2ffd700 1 -- 172.21.15.194:0/265558600 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f60ac014420 con 0x7f60bc0f6510 2024-01-27T23:32:32.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.095+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f60bc0fbf60 con 0x7f60bc0f6510 2024-01-27T23:32:32.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.096+0000 7f60b2ffd700 1 -- 172.21.15.194:0/265558600 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f60ac01d7f0 con 0x7f60bc0f6510 2024-01-27T23:32:32.091 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:32.091 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f60a802ff60 msgr2=0x7f60a8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f60a802ff60 0x7f60a8032420 secure :-1 s=READY pgs=156 cs=0 l=1 rev1=1 rx=0x7f60b80096a0 tx=0x7f60b8006b40).stop 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 msgr2=0x7f60bc106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc106730 secure :-1 s=READY pgs=380 cs=0 l=1 rev1=1 rx=0x7f60ac013510 tx=0x7f60ac00b040).stop 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 shutdown_connections 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f60a802ff60 0x7f60a8032420 unknown :-1 s=CLOSED pgs=156 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 --2- 172.21.15.194:0/265558600 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f60bc0f6510 0x7f60bc106730 unknown :-1 s=CLOSED pgs=380 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.097+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 >> 172.21.15.194:0/265558600 conn(0x7f60bc0f1c70 msgr2=0x7f60bc0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.098+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 shutdown_connections 2024-01-27T23:32:32.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:32.098+0000 7f60c47cd700 1 -- 172.21.15.194:0/265558600 wait complete. 2024-01-27T23:32:32.093 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:32.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:31.324863+0000 mgr.smithi121.hcppvz (mgr.14162) 223 : cluster [DBG] pgmap v211: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:32.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:32.097280+0000 mon.smithi121 (mon.0) 286 : audit [DBG] from='client.? 172.21.15.194:0/265558600' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:33.456 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:33.456 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:34.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 -- 172.21.15.194:0/3241420225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 msgr2=0x7fcbdc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:34.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 --2- 172.21.15.194:0/3241420225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc0f5830 secure :-1 s=READY pgs=381 cs=0 l=1 rev1=1 rx=0x7fcbd0009fb0 tx=0x7fcbd0009b30).stop 2024-01-27T23:32:34.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 -- 172.21.15.194:0/3241420225 shutdown_connections 2024-01-27T23:32:34.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 --2- 172.21.15.194:0/3241420225 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc0f5830 unknown :-1 s=CLOSED pgs=381 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:34.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 -- 172.21.15.194:0/3241420225 >> 172.21.15.194:0/3241420225 conn(0x7fcbdc0f0b70 msgr2=0x7fcbdc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:34.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 -- 172.21.15.194:0/3241420225 shutdown_connections 2024-01-27T23:32:34.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 -- 172.21.15.194:0/3241420225 wait complete. 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 Processor -- start 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.606+0000 7fcbe4711700 1 -- start start 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe4711700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc105630 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe4711700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fcbdc105b20 con 0x7fcbdc0f5410 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe24ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc105630 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe24ad700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc105630 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:59204/0 (socket says 172.21.15.194:59204) 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe24ad700 1 -- 172.21.15.194:0/2606628830 learned_addr learned my addr 172.21.15.194:0/2606628830 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe24ad700 1 -- 172.21.15.194:0/2606628830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fcbd0005040 con 0x7fcbdc0f5410 2024-01-27T23:32:34.602 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe24ad700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc105630 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcbd0014000 tx=0x7fcbd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:34.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbceffd700 1 -- 172.21.15.194:0/2606628830 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbd00095f0 con 0x7fcbdc0f5410 2024-01-27T23:32:34.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fcbdc105d20 con 0x7fcbdc0f5410 2024-01-27T23:32:34.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.607+0000 7fcbceffd700 1 -- 172.21.15.194:0/2606628830 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fcbd0007e70 con 0x7fcbdc0f5410 2024-01-27T23:32:34.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.608+0000 7fcbceffd700 1 -- 172.21.15.194:0/2606628830 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fcbd0014ea0 con 0x7fcbdc0f5410 2024-01-27T23:32:34.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.608+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fcbdc1061b0 con 0x7fcbdc0f5410 2024-01-27T23:32:34.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.608+0000 7fcbceffd700 1 -- 172.21.15.194:0/2606628830 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fcbd0025070 con 0x7fcbdc0f5410 2024-01-27T23:32:34.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.608+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fcbdc045ad0 con 0x7fcbdc0f5410 2024-01-27T23:32:34.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.608+0000 7fcbceffd700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcbc802ff60 0x7fcbc8032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:34.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.609+0000 7fcbceffd700 1 -- 172.21.15.194:0/2606628830 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fcbd0040970 con 0x7fcbdc0f5410 2024-01-27T23:32:34.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.609+0000 7fcbe1cac700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcbc802ff60 0x7fcbc8032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:34.604 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.609+0000 7fcbe1cac700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcbc802ff60 0x7fcbc8032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fcbd80096a0 tx=0x7fcbd8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:34.607 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.613+0000 7fcbceffd700 1 -- 172.21.15.194:0/2606628830 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fcbd00079c0 con 0x7fcbdc0f5410 2024-01-27T23:32:34.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.821+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fcbdc0fae60 con 0x7fcbdc0f5410 2024-01-27T23:32:34.815 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.821+0000 7fcbceffd700 1 -- 172.21.15.194:0/2606628830 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fcbd000f300 con 0x7fcbdc0f5410 2024-01-27T23:32:34.816 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:34.816 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:34.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcbc802ff60 msgr2=0x7fcbc8032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:34.817 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcbc802ff60 0x7fcbc8032420 secure :-1 s=READY pgs=157 cs=0 l=1 rev1=1 rx=0x7fcbd80096a0 tx=0x7fcbd8006b40).stop 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 msgr2=0x7fcbdc105630 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc105630 secure :-1 s=READY pgs=382 cs=0 l=1 rev1=1 rx=0x7fcbd0014000 tx=0x7fcbd000b040).stop 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 shutdown_connections 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fcbc802ff60 0x7fcbc8032420 unknown :-1 s=CLOSED pgs=157 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 --2- 172.21.15.194:0/2606628830 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fcbdc0f5410 0x7fcbdc105630 unknown :-1 s=CLOSED pgs=382 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 >> 172.21.15.194:0/2606628830 conn(0x7fcbdc0f0b70 msgr2=0x7fcbdc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 shutdown_connections 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:34.823+0000 7fcbe4711700 1 -- 172.21.15.194:0/2606628830 wait complete. 2024-01-27T23:32:34.818 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:34.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:33.325286+0000 mgr.smithi121.hcppvz (mgr.14162) 224 : cluster [DBG] pgmap v212: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:35.695 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:35 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:34.822843+0000 mon.smithi121 (mon.0) 287 : audit [DBG] from='client.? 172.21.15.194:0/2606628830' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:36.185 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:36.186 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:36.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:35.325578+0000 mgr.smithi121.hcppvz (mgr.14162) 225 : cluster [DBG] pgmap v213: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:37.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.341+0000 7f53a7e48700 1 -- 172.21.15.194:0/3371941788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 msgr2=0x7f53a00f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.341+0000 7f53a7e48700 1 --2- 172.21.15.194:0/3371941788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a00f9190 secure :-1 s=READY pgs=383 cs=0 l=1 rev1=1 rx=0x7f5398009fb0 tx=0x7f5398009b30).stop 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.341+0000 7f53a7e48700 1 -- 172.21.15.194:0/3371941788 shutdown_connections 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.341+0000 7f53a7e48700 1 --2- 172.21.15.194:0/3371941788 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a00f9190 unknown :-1 s=CLOSED pgs=383 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.341+0000 7f53a7e48700 1 -- 172.21.15.194:0/3371941788 >> 172.21.15.194:0/3371941788 conn(0x7f53a00f0b70 msgr2=0x7f53a00f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.341+0000 7f53a7e48700 1 -- 172.21.15.194:0/3371941788 shutdown_connections 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.341+0000 7f53a7e48700 1 -- 172.21.15.194:0/3371941788 wait complete. 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.342+0000 7f53a7e48700 1 Processor -- start 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.342+0000 7f53a7e48700 1 -- start start 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.342+0000 7f53a7e48700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a01066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.342+0000 7f53a7e48700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f53a0106bd0 con 0x7f53a00601e0 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.342+0000 7f53a5be4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a01066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:37.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.342+0000 7f53a5be4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a01066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:58004/0 (socket says 172.21.15.194:58004) 2024-01-27T23:32:37.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.342+0000 7f53a5be4700 1 -- 172.21.15.194:0/2096677253 learned_addr learned my addr 172.21.15.194:0/2096677253 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:37.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.343+0000 7f53a5be4700 1 -- 172.21.15.194:0/2096677253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5398005040 con 0x7f53a00601e0 2024-01-27T23:32:37.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.343+0000 7f53a5be4700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a01066e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f5398004dd0 tx=0x7f5398009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:37.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.343+0000 7f539e7fc700 1 -- 172.21.15.194:0/2096677253 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5398018070 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.343+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f53a0106dd0 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.343+0000 7f539e7fc700 1 -- 172.21.15.194:0/2096677253 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5398013410 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.343+0000 7f539e7fc700 1 -- 172.21.15.194:0/2096677253 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f539801c3c0 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.343+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f53a0107260 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.344+0000 7f539e7fc700 1 -- 172.21.15.194:0/2096677253 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f539801d420 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.344+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f53a0045ad0 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.344+0000 7f539e7fc700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f538802ff60 0x7f5388032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.344+0000 7f539e7fc700 1 -- 172.21.15.194:0/2096677253 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5398041330 con 0x7f53a00601e0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.344+0000 7f53a53e3700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f538802ff60 0x7f5388032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:37.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.345+0000 7f53a53e3700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f538802ff60 0x7f5388032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f53940096a0 tx=0x7f5394006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:37.343 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.349+0000 7f539e7fc700 1 -- 172.21.15.194:0/2096677253 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f539801d820 con 0x7f53a00601e0 2024-01-27T23:32:37.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.555+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f53a0108110 con 0x7f53a00601e0 2024-01-27T23:32:37.550 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.556+0000 7f539e7fc700 1 -- 172.21.15.194:0/2096677253 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f539801b370 con 0x7f53a00601e0 2024-01-27T23:32:37.550 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:37.550 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f538802ff60 msgr2=0x7f5388032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f538802ff60 0x7f5388032420 secure :-1 s=READY pgs=158 cs=0 l=1 rev1=1 rx=0x7f53940096a0 tx=0x7f5394006b40).stop 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 msgr2=0x7f53a01066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a01066e0 secure :-1 s=READY pgs=384 cs=0 l=1 rev1=1 rx=0x7f5398004dd0 tx=0x7f5398009b30).stop 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 shutdown_connections 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f538802ff60 0x7f5388032420 unknown :-1 s=CLOSED pgs=158 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 --2- 172.21.15.194:0/2096677253 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f53a00601e0 0x7f53a01066e0 unknown :-1 s=CLOSED pgs=384 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 >> 172.21.15.194:0/2096677253 conn(0x7f53a00f0b70 msgr2=0x7f53a00f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 shutdown_connections 2024-01-27T23:32:37.552 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:37.557+0000 7f53a7e48700 1 -- 172.21.15.194:0/2096677253 wait complete. 2024-01-27T23:32:37.553 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:38.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:37.325965+0000 mgr.smithi121.hcppvz (mgr.14162) 226 : cluster [DBG] pgmap v214: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:38.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:37.556925+0000 mon.smithi121 (mon.0) 288 : audit [DBG] from='client.? 172.21.15.194:0/2096677253' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:38.954 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:38.955 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 -- 172.21.15.194:0/2306681338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 msgr2=0x7fc1080f90f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 --2- 172.21.15.194:0/2306681338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1080f90f0 secure :-1 s=READY pgs=385 cs=0 l=1 rev1=1 rx=0x7fc0f8009fb0 tx=0x7fc0f8009b30).stop 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 -- 172.21.15.194:0/2306681338 shutdown_connections 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 --2- 172.21.15.194:0/2306681338 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1080f90f0 unknown :-1 s=CLOSED pgs=385 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 -- 172.21.15.194:0/2306681338 >> 172.21.15.194:0/2306681338 conn(0x7fc1080f0b70 msgr2=0x7fc1080f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 -- 172.21.15.194:0/2306681338 shutdown_connections 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 -- 172.21.15.194:0/2306681338 wait complete. 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 Processor -- start 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.088+0000 7fc10e338700 1 -- start start 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc10e338700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1081066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:40.083 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc10e338700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc108106bd0 con 0x7fc108060280 2024-01-27T23:32:40.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc107fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1081066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:40.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc107fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1081066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:58022/0 (socket says 172.21.15.194:58022) 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc107fff700 1 -- 172.21.15.194:0/717140097 learned_addr learned my addr 172.21.15.194:0/717140097 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc107fff700 1 -- 172.21.15.194:0/717140097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc0f8005040 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc107fff700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1081066e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc0f8013510 tx=0x7fc0f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc104ff9700 1 -- 172.21.15.194:0/717140097 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0f8009320 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc104ff9700 1 -- 172.21.15.194:0/717140097 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc0f8013f00 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc104ff9700 1 -- 172.21.15.194:0/717140097 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc0f8014dc0 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc108106dd0 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.089+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc108107260 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.090+0000 7fc104ff9700 1 -- 172.21.15.194:0/717140097 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc0f80095f0 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.090+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc108045ad0 con 0x7fc108060280 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.090+0000 7fc104ff9700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc0f002ffb0 0x7fc0f0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:40.086 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.090+0000 7fc104ff9700 1 -- 172.21.15.194:0/717140097 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc0f80410b0 con 0x7fc108060280 2024-01-27T23:32:40.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.091+0000 7fc1077fe700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc0f002ffb0 0x7fc0f0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:40.087 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.091+0000 7fc1077fe700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc0f002ffb0 0x7fc0f0032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fc0fc0096a0 tx=0x7fc0fc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:40.089 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.095+0000 7fc104ff9700 1 -- 172.21.15.194:0/717140097 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc0f806d2b0 con 0x7fc108060280 2024-01-27T23:32:40.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.301+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc108108110 con 0x7fc108060280 2024-01-27T23:32:40.296 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.302+0000 7fc104ff9700 1 -- 172.21.15.194:0/717140097 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc0f8014420 con 0x7fc108060280 2024-01-27T23:32:40.296 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:40.297 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:40.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.303+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc0f002ffb0 msgr2=0x7fc0f0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:40.299 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc0f002ffb0 0x7fc0f0032470 secure :-1 s=READY pgs=159 cs=0 l=1 rev1=1 rx=0x7fc0fc0096a0 tx=0x7fc0fc006b40).stop 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 msgr2=0x7fc1081066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1081066e0 secure :-1 s=READY pgs=386 cs=0 l=1 rev1=1 rx=0x7fc0f8013510 tx=0x7fc0f800b040).stop 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 shutdown_connections 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc0f002ffb0 0x7fc0f0032470 unknown :-1 s=CLOSED pgs=159 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 --2- 172.21.15.194:0/717140097 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc108060280 0x7fc1081066e0 unknown :-1 s=CLOSED pgs=386 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 >> 172.21.15.194:0/717140097 conn(0x7fc1080f0b70 msgr2=0x7fc1080f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 shutdown_connections 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:40.304+0000 7fc10e338700 1 -- 172.21.15.194:0/717140097 wait complete. 2024-01-27T23:32:40.300 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:40.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:39.326420+0000 mgr.smithi121.hcppvz (mgr.14162) 227 : cluster [DBG] pgmap v215: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:40.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:40.303140+0000 mon.smithi121 (mon.0) 289 : audit [DBG] from='client.? 172.21.15.194:0/717140097' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:41.689 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:41.689 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:42.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:41.326716+0000 mgr.smithi121.hcppvz (mgr.14162) 228 : cluster [DBG] pgmap v216: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:42.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.873+0000 7f186f4bc700 1 -- 172.21.15.194:0/3821517886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 msgr2=0x7f18680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:42.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.873+0000 7f186f4bc700 1 --2- 172.21.15.194:0/3821517886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680f5830 secure :-1 s=READY pgs=387 cs=0 l=1 rev1=1 rx=0x7f1858009fb0 tx=0x7f1858009b30).stop 2024-01-27T23:32:42.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.873+0000 7f186f4bc700 1 -- 172.21.15.194:0/3821517886 shutdown_connections 2024-01-27T23:32:42.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.873+0000 7f186f4bc700 1 --2- 172.21.15.194:0/3821517886 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680f5830 unknown :-1 s=CLOSED pgs=387 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:42.869 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.873+0000 7f186f4bc700 1 -- 172.21.15.194:0/3821517886 >> 172.21.15.194:0/3821517886 conn(0x7f18680f0b70 msgr2=0x7f18680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.873+0000 7f186f4bc700 1 -- 172.21.15.194:0/3821517886 shutdown_connections 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.873+0000 7f186f4bc700 1 -- 172.21.15.194:0/3821517886 wait complete. 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186f4bc700 1 Processor -- start 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186f4bc700 1 -- start start 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186f4bc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186f4bc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f18680fb820 con 0x7f18680f5410 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186d258700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186d258700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:58054/0 (socket says 172.21.15.194:58054) 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186d258700 1 -- 172.21.15.194:0/3227004299 learned_addr learned my addr 172.21.15.194:0/3227004299 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186d258700 1 -- 172.21.15.194:0/3227004299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1858005040 con 0x7f18680f5410 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.874+0000 7f186d258700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680fd3a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f1858013510 tx=0x7f185800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.875+0000 7f185dffb700 1 -- 172.21.15.194:0/3227004299 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1858009320 con 0x7f18680f5410 2024-01-27T23:32:42.870 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.875+0000 7f185dffb700 1 -- 172.21.15.194:0/3227004299 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1858013f00 con 0x7f18680f5410 2024-01-27T23:32:42.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.875+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f18680fb9c0 con 0x7f18680f5410 2024-01-27T23:32:42.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.875+0000 7f185dffb700 1 -- 172.21.15.194:0/3227004299 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1858014d90 con 0x7f18680f5410 2024-01-27T23:32:42.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.875+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f18680fbe50 con 0x7f18680f5410 2024-01-27T23:32:42.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.876+0000 7f185dffb700 1 -- 172.21.15.194:0/3227004299 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f18580095f0 con 0x7f18680f5410 2024-01-27T23:32:42.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.876+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1868059100 con 0x7f18680f5410 2024-01-27T23:32:42.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.876+0000 7f185dffb700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f185402ff60 0x7f1854032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:42.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.876+0000 7f185dffb700 1 -- 172.21.15.194:0/3227004299 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1858041080 con 0x7f18680f5410 2024-01-27T23:32:42.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.876+0000 7f186ca57700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f185402ff60 0x7f1854032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:42.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.876+0000 7f186ca57700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f185402ff60 0x7f1854032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f18640096a0 tx=0x7f1864006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:42.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:42.881+0000 7f185dffb700 1 -- 172.21.15.194:0/3227004299 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1858014420 con 0x7f18680f5410 2024-01-27T23:32:43.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.087+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1868045ad0 con 0x7f18680f5410 2024-01-27T23:32:43.082 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.088+0000 7f185dffb700 1 -- 172.21.15.194:0/3227004299 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1858010230 con 0x7f18680f5410 2024-01-27T23:32:43.083 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:43.083 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f185402ff60 msgr2=0x7f1854032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f185402ff60 0x7f1854032420 secure :-1 s=READY pgs=160 cs=0 l=1 rev1=1 rx=0x7f18640096a0 tx=0x7f1864006b40).stop 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 msgr2=0x7f18680fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680fd3a0 secure :-1 s=READY pgs=388 cs=0 l=1 rev1=1 rx=0x7f1858013510 tx=0x7f185800b040).stop 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 shutdown_connections 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f185402ff60 0x7f1854032420 unknown :-1 s=CLOSED pgs=160 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 --2- 172.21.15.194:0/3227004299 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f18680f5410 0x7f18680fd3a0 unknown :-1 s=CLOSED pgs=388 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 >> 172.21.15.194:0/3227004299 conn(0x7f18680f0b70 msgr2=0x7f186805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 shutdown_connections 2024-01-27T23:32:43.085 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:43.090+0000 7f186f4bc700 1 -- 172.21.15.194:0/3227004299 wait complete. 2024-01-27T23:32:43.086 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:43.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:43 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:43.089470+0000 mon.smithi121 (mon.0) 290 : audit [DBG] from='client.? 172.21.15.194:0/3227004299' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:44.460 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:44.460 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:44.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:43.327067+0000 mgr.smithi121.hcppvz (mgr.14162) 229 : cluster [DBG] pgmap v217: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:44.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:32:43.887962+0000 mgr.smithi121.hcppvz (mgr.14162) 230 : cephadm [ERR] cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi121 ... 2024-01-27T23:32:44.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Verifying port 9100 ... 2024-01-27T23:32:44.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Non-zero exit code 1 from systemctl start ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121 2024-01-27T23:32:44.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: systemctl: stderr Job for ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121.service failed because a timeout was exceeded. 2024-01-27T23:32:44.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: systemctl: stderr See "systemctl status ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121.service" and "journalctl -xe" for details. 2024-01-27T23:32:44.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Traceback (most recent call last): 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: main() 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: r = ctx.func(ctx) 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: return func(ctx) 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ports=daemon_ports) 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-27T23:32:44.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: RuntimeError: Failed command: systemctl start ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Traceback (most recent call last): 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1144, in _remote_connection 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: yield (conn, connr) 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/usr/share/ceph/mgr/cephadm/serve.py", line 1059, in _run_cephadm 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: code, '\n'.join(err))) 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: orchestrator._interface.OrchestratorError: cephadm exited with an error code: 1, stderr:Deploy daemon node-exporter.smithi121 ... 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Verifying port 9100 ... 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Non-zero exit code 1 from systemctl start ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: systemctl: stderr Job for ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121.service failed because a timeout was exceeded. 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: systemctl: stderr See "systemctl status ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121.service" and "journalctl -xe" for details. 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: Traceback (most recent call last): 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8029, in 2024-01-27T23:32:44.594 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: main() 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 8017, in main 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: r = ctx.func(ctx) 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1760, in _default_image 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: return func(ctx) 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 4267, in command_deploy 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: ports=daemon_ports) 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2608, in deploy_daemon 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: c, osd_fsid=osd_fsid, ports=ports) 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 2806, in deploy_daemon_units 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: call_throws(ctx, ['systemctl', 'start', unit_name]) 2024-01-27T23:32:44.595 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: File "/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/cephadm.30cb78bdbbafb384af862e1c2292b944f15942b586128e91262b43e91e11ae90", line 1464, in call_throws 2024-01-27T23:32:44.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: raise RuntimeError('Failed command: %s' % ' '.join(command)) 2024-01-27T23:32:44.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: RuntimeError: Failed command: systemctl start ceph-2005e414-bd6b-11ee-95b2-87774f69a715@node-exporter.smithi121 2024-01-27T23:32:44.596 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cephadm 2024-01-27T23:32:43.893957+0000 mgr.smithi121.hcppvz (mgr.14162) 231 : cephadm [INF] Deploying daemon prometheus.smithi121 on smithi121 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.547+0000 7f32fee5a700 1 -- 172.21.15.194:0/4022155721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 msgr2=0x7f32f80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.547+0000 7f32fee5a700 1 --2- 172.21.15.194:0/4022155721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f80f5830 secure :-1 s=READY pgs=389 cs=0 l=1 rev1=1 rx=0x7f32ec009fb0 tx=0x7f32ec009b30).stop 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.547+0000 7f32fee5a700 1 -- 172.21.15.194:0/4022155721 shutdown_connections 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.547+0000 7f32fee5a700 1 --2- 172.21.15.194:0/4022155721 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f80f5830 unknown :-1 s=CLOSED pgs=389 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.547+0000 7f32fee5a700 1 -- 172.21.15.194:0/4022155721 >> 172.21.15.194:0/4022155721 conn(0x7f32f80f0b70 msgr2=0x7f32f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.547+0000 7f32fee5a700 1 -- 172.21.15.194:0/4022155721 shutdown_connections 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.547+0000 7f32fee5a700 1 -- 172.21.15.194:0/4022155721 wait complete. 2024-01-27T23:32:45.542 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.548+0000 7f32fee5a700 1 Processor -- start 2024-01-27T23:32:45.543 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.548+0000 7f32fee5a700 1 -- start start 2024-01-27T23:32:45.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.548+0000 7f32fee5a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f8109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:45.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.548+0000 7f32fee5a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f32f8060240 con 0x7f32f80f5410 2024-01-27T23:32:45.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.548+0000 7f32fcbf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f8109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:45.544 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.548+0000 7f32fcbf6700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f8109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:51258/0 (socket says 172.21.15.194:51258) 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.548+0000 7f32fcbf6700 1 -- 172.21.15.194:0/468119670 learned_addr learned my addr 172.21.15.194:0/468119670 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.549+0000 7f32fcbf6700 1 -- 172.21.15.194:0/468119670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f32ec005040 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.549+0000 7f32fcbf6700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f8109430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f32ec013510 tx=0x7f32ec00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.549+0000 7f32f57fa700 1 -- 172.21.15.194:0/468119670 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32ec009320 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.549+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f32f8060440 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.549+0000 7f32f57fa700 1 -- 172.21.15.194:0/468119670 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f32ec013f00 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.549+0000 7f32f57fa700 1 -- 172.21.15.194:0/468119670 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f32ec014d90 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.549+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f32f8109ae0 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.550+0000 7f32f57fa700 1 -- 172.21.15.194:0/468119670 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f32ec014420 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.550+0000 7f32f57fa700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f32e802ffb0 0x7f32e8032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.550+0000 7f32f57fa700 1 -- 172.21.15.194:0/468119670 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f32ec041070 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.550+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f32f8059100 con 0x7f32f80f5410 2024-01-27T23:32:45.545 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.550+0000 7f32f7fff700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f32e802ffb0 0x7f32e8032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:45.546 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.551+0000 7f32f7fff700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f32e802ffb0 0x7f32e8032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f32e40096a0 tx=0x7f32e4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:45.549 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.555+0000 7f32f57fa700 1 -- 172.21.15.194:0/468119670 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f32ec0095f0 con 0x7f32f80f5410 2024-01-27T23:32:45.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.758+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f32f8045ad0 con 0x7f32f80f5410 2024-01-27T23:32:45.753 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.759+0000 7f32f57fa700 1 -- 172.21.15.194:0/468119670 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f32ec01db70 con 0x7f32f80f5410 2024-01-27T23:32:45.753 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:45.753 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f32e802ffb0 msgr2=0x7f32e8032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f32e802ffb0 0x7f32e8032470 secure :-1 s=READY pgs=161 cs=0 l=1 rev1=1 rx=0x7f32e40096a0 tx=0x7f32e4006b40).stop 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 msgr2=0x7f32f8109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f8109430 secure :-1 s=READY pgs=390 cs=0 l=1 rev1=1 rx=0x7f32ec013510 tx=0x7f32ec00b040).stop 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 shutdown_connections 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f32e802ffb0 0x7f32e8032470 unknown :-1 s=CLOSED pgs=161 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 --2- 172.21.15.194:0/468119670 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f32f80f5410 0x7f32f8109430 unknown :-1 s=CLOSED pgs=390 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 >> 172.21.15.194:0/468119670 conn(0x7f32f80f0b70 msgr2=0x7f32f80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:45.755 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 shutdown_connections 2024-01-27T23:32:45.756 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:45.760+0000 7f32fee5a700 1 -- 172.21.15.194:0/468119670 wait complete. 2024-01-27T23:32:45.756 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:46.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:45.327385+0000 mgr.smithi121.hcppvz (mgr.14162) 232 : cluster [DBG] pgmap v218: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:46.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:45.760109+0000 mon.smithi121 (mon.0) 291 : audit [DBG] from='client.? 172.21.15.194:0/468119670' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:47.132 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:47.132 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:48.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.362+0000 7f899d9fc700 1 -- 172.21.15.194:0/1523617113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 msgr2=0x7f89980f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:48.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.362+0000 7f899d9fc700 1 --2- 172.21.15.194:0/1523617113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89980f5830 secure :-1 s=READY pgs=391 cs=0 l=1 rev1=1 rx=0x7f898c009fb0 tx=0x7f898c009b30).stop 2024-01-27T23:32:48.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.362+0000 7f899d9fc700 1 -- 172.21.15.194:0/1523617113 shutdown_connections 2024-01-27T23:32:48.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.362+0000 7f899d9fc700 1 --2- 172.21.15.194:0/1523617113 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89980f5830 unknown :-1 s=CLOSED pgs=391 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:48.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.362+0000 7f899d9fc700 1 -- 172.21.15.194:0/1523617113 >> 172.21.15.194:0/1523617113 conn(0x7f89980f0b70 msgr2=0x7f89980f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:48.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.362+0000 7f899d9fc700 1 -- 172.21.15.194:0/1523617113 shutdown_connections 2024-01-27T23:32:48.357 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.362+0000 7f899d9fc700 1 -- 172.21.15.194:0/1523617113 wait complete. 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.363+0000 7f899d9fc700 1 Processor -- start 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.363+0000 7f899d9fc700 1 -- start start 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.363+0000 7f899d9fc700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89981066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.363+0000 7f899d9fc700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8998106bd0 con 0x7f89980f5410 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.363+0000 7f8996ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89981066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.363+0000 7f8996ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89981066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:51288/0 (socket says 172.21.15.194:51288) 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.363+0000 7f8996ffd700 1 -- 172.21.15.194:0/3668274749 learned_addr learned my addr 172.21.15.194:0/3668274749 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.364+0000 7f8996ffd700 1 -- 172.21.15.194:0/3668274749 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f898c005040 con 0x7f89980f5410 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.364+0000 7f8996ffd700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89981066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f898c013510 tx=0x7f898c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:48.360 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.364+0000 7f897f7fe700 1 -- 172.21.15.194:0/3668274749 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c009320 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.364+0000 7f897f7fe700 1 -- 172.21.15.194:0/3668274749 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f898c013f00 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.364+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8998106dd0 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.364+0000 7f897f7fe700 1 -- 172.21.15.194:0/3668274749 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f898c014d90 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.364+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8998107260 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.365+0000 7f897f7fe700 1 -- 172.21.15.194:0/3668274749 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f898c0263a0 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.365+0000 7f897f7fe700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f898002ff60 0x7f8980032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.365+0000 7f897f7fe700 1 -- 172.21.15.194:0/3668274749 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f898c0410d0 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.365+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8998045ad0 con 0x7f89980f5410 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.365+0000 7f89967fc700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f898002ff60 0x7f8980032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:48.361 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.366+0000 7f89967fc700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f898002ff60 0x7f8980032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f89880096a0 tx=0x7f8988006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:48.364 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.370+0000 7f897f7fe700 1 -- 172.21.15.194:0/3668274749 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f898c012920 con 0x7f89980f5410 2024-01-27T23:32:48.570 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.576+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8998107fb0 con 0x7f89980f5410 2024-01-27T23:32:48.571 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.577+0000 7f897f7fe700 1 -- 172.21.15.194:0/3668274749 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f898c02b020 con 0x7f89980f5410 2024-01-27T23:32:48.571 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:48.572 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:48.573 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f898002ff60 msgr2=0x7f8980032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:48.573 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f898002ff60 0x7f8980032420 secure :-1 s=READY pgs=162 cs=0 l=1 rev1=1 rx=0x7f89880096a0 tx=0x7f8988006b40).stop 2024-01-27T23:32:48.573 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 msgr2=0x7f89981066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:48.573 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89981066e0 secure :-1 s=READY pgs=392 cs=0 l=1 rev1=1 rx=0x7f898c013510 tx=0x7f898c00b040).stop 2024-01-27T23:32:48.573 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 shutdown_connections 2024-01-27T23:32:48.573 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f898002ff60 0x7f8980032420 unknown :-1 s=CLOSED pgs=162 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:48.574 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 --2- 172.21.15.194:0/3668274749 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f89980f5410 0x7f89981066e0 unknown :-1 s=CLOSED pgs=392 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:48.574 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.578+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 >> 172.21.15.194:0/3668274749 conn(0x7f89980f0b70 msgr2=0x7f899805fe00 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:48.574 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.579+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 shutdown_connections 2024-01-27T23:32:48.574 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:48.579+0000 7f899d9fc700 1 -- 172.21.15.194:0/3668274749 wait complete. 2024-01-27T23:32:48.574 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:48.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:47.327715+0000 mgr.smithi121.hcppvz (mgr.14162) 233 : cluster [DBG] pgmap v219: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:49.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:49 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:48.578034+0000 mon.smithi121 (mon.0) 292 : audit [DBG] from='client.? 172.21.15.194:0/3668274749' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:49.946 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:49.947 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:50.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:49.328107+0000 mgr.smithi121.hcppvz (mgr.14162) 234 : cluster [DBG] pgmap v220: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 -- 172.21.15.194:0/2129636608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 msgr2=0x7f6a440f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 --2- 172.21.15.194:0/2129636608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a440f66c0 secure :-1 s=READY pgs=393 cs=0 l=1 rev1=1 rx=0x7f6a40009fb0 tx=0x7f6a40009b30).stop 2024-01-27T23:32:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 -- 172.21.15.194:0/2129636608 shutdown_connections 2024-01-27T23:32:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 --2- 172.21.15.194:0/2129636608 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a440f66c0 unknown :-1 s=CLOSED pgs=393 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 -- 172.21.15.194:0/2129636608 >> 172.21.15.194:0/2129636608 conn(0x7f6a440f1aa0 msgr2=0x7f6a440f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 -- 172.21.15.194:0/2129636608 shutdown_connections 2024-01-27T23:32:51.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 -- 172.21.15.194:0/2129636608 wait complete. 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.131+0000 7f6a4b578700 1 Processor -- start 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4b578700 1 -- start start 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4b578700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a441064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4b578700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6a441069a0 con 0x7f6a440f62a0 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4a576700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a441064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4a576700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a441064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:51316/0 (socket says 172.21.15.194:51316) 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4a576700 1 -- 172.21.15.194:0/3892412351 learned_addr learned my addr 172.21.15.194:0/3892412351 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4a576700 1 -- 172.21.15.194:0/3892412351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6a40005040 con 0x7f6a440f62a0 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.132+0000 7f6a4a576700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a441064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f6a40013510 tx=0x7f6a4000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.133+0000 7f6a3affd700 1 -- 172.21.15.194:0/3892412351 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a40009320 con 0x7f6a440f62a0 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.133+0000 7f6a3affd700 1 -- 172.21.15.194:0/3892412351 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6a40013f00 con 0x7f6a440f62a0 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.133+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6a44106ba0 con 0x7f6a440f62a0 2024-01-27T23:32:51.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.133+0000 7f6a3affd700 1 -- 172.21.15.194:0/3892412351 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6a40014dc0 con 0x7f6a440f62a0 2024-01-27T23:32:51.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.133+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f6a44107030 con 0x7f6a440f62a0 2024-01-27T23:32:51.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.133+0000 7f6a3affd700 1 -- 172.21.15.194:0/3892412351 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f6a400095f0 con 0x7f6a440f62a0 2024-01-27T23:32:51.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.133+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6a44045ad0 con 0x7f6a440f62a0 2024-01-27T23:32:51.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.134+0000 7f6a3affd700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6a3002ff60 0x7f6a30032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:51.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.134+0000 7f6a3affd700 1 -- 172.21.15.194:0/3892412351 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6a40041040 con 0x7f6a440f62a0 2024-01-27T23:32:51.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.134+0000 7f6a49d75700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6a3002ff60 0x7f6a30032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:51.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.134+0000 7f6a49d75700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6a3002ff60 0x7f6a30032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6a340096a0 tx=0x7f6a34006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:51.132 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.138+0000 7f6a3affd700 1 -- 172.21.15.194:0/3892412351 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f6a40014420 con 0x7f6a440f62a0 2024-01-27T23:32:51.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.342+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f6a44107ee0 con 0x7f6a440f62a0 2024-01-27T23:32:51.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.343+0000 7f6a3affd700 1 -- 172.21.15.194:0/3892412351 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6a40010230 con 0x7f6a440f62a0 2024-01-27T23:32:51.338 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:51.338 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:51.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.344+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6a3002ff60 msgr2=0x7f6a30032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:51.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.344+0000 7f6a4b578700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6a3002ff60 0x7f6a30032420 secure :-1 s=READY pgs=163 cs=0 l=1 rev1=1 rx=0x7f6a340096a0 tx=0x7f6a34006b40).stop 2024-01-27T23:32:51.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.344+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 msgr2=0x7f6a441064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:51.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.344+0000 7f6a4b578700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a441064b0 secure :-1 s=READY pgs=394 cs=0 l=1 rev1=1 rx=0x7f6a40013510 tx=0x7f6a4000b040).stop 2024-01-27T23:32:51.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.345+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 shutdown_connections 2024-01-27T23:32:51.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.345+0000 7f6a4b578700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f6a3002ff60 0x7f6a30032420 unknown :-1 s=CLOSED pgs=163 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:51.339 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.345+0000 7f6a4b578700 1 --2- 172.21.15.194:0/3892412351 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f6a440f62a0 0x7f6a441064b0 unknown :-1 s=CLOSED pgs=394 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:51.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.345+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 >> 172.21.15.194:0/3892412351 conn(0x7f6a440f1aa0 msgr2=0x7f6a440f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:51.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.345+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 shutdown_connections 2024-01-27T23:32:51.340 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:51.345+0000 7f6a4b578700 1 -- 172.21.15.194:0/3892412351 wait complete. 2024-01-27T23:32:51.340 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:52.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:51.328491+0000 mgr.smithi121.hcppvz (mgr.14162) 235 : cluster [DBG] pgmap v221: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:52.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:51.344338+0000 mon.smithi121 (mon.0) 293 : audit [DBG] from='client.? 172.21.15.194:0/3892412351' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:52.739 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:52.739 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:53.875 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.879+0000 7fb6e77b5700 1 -- 172.21.15.194:0/2305818310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 msgr2=0x7fb6e00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.879+0000 7fb6e77b5700 1 --2- 172.21.15.194:0/2305818310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e00f6930 secure :-1 s=READY pgs=395 cs=0 l=1 rev1=1 rx=0x7fb6d4009fb0 tx=0x7fb6d4009b30).stop 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.879+0000 7fb6e77b5700 1 -- 172.21.15.194:0/2305818310 shutdown_connections 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.879+0000 7fb6e77b5700 1 --2- 172.21.15.194:0/2305818310 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e00f6930 unknown :-1 s=CLOSED pgs=395 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.879+0000 7fb6e77b5700 1 -- 172.21.15.194:0/2305818310 >> 172.21.15.194:0/2305818310 conn(0x7fb6e00f1c70 msgr2=0x7fb6e00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.879+0000 7fb6e77b5700 1 -- 172.21.15.194:0/2305818310 shutdown_connections 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.879+0000 7fb6e77b5700 1 -- 172.21.15.194:0/2305818310 wait complete. 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.880+0000 7fb6e77b5700 1 Processor -- start 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.880+0000 7fb6e77b5700 1 -- start start 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.880+0000 7fb6e77b5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.880+0000 7fb6e77b5700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb6e0106c20 con 0x7fb6e00f6510 2024-01-27T23:32:53.876 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.880+0000 7fb6e5551700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.880+0000 7fb6e5551700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:51320/0 (socket says 172.21.15.194:51320) 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.880+0000 7fb6e5551700 1 -- 172.21.15.194:0/3734685520 learned_addr learned my addr 172.21.15.194:0/3734685520 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.881+0000 7fb6e5551700 1 -- 172.21.15.194:0/3734685520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb6d4005040 con 0x7fb6e00f6510 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.881+0000 7fb6e5551700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e0106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb6d4013510 tx=0x7fb6d400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.881+0000 7fb6d1ffb700 1 -- 172.21.15.194:0/3734685520 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6d4009320 con 0x7fb6e00f6510 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.881+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb6e0106e20 con 0x7fb6e00f6510 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.881+0000 7fb6d1ffb700 1 -- 172.21.15.194:0/3734685520 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb6d4013f00 con 0x7fb6e00f6510 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.881+0000 7fb6d1ffb700 1 -- 172.21.15.194:0/3734685520 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb6d4014d90 con 0x7fb6e00f6510 2024-01-27T23:32:53.877 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.881+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb6e01072b0 con 0x7fb6e00f6510 2024-01-27T23:32:53.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.882+0000 7fb6d1ffb700 1 -- 172.21.15.194:0/3734685520 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb6d4014420 con 0x7fb6e00f6510 2024-01-27T23:32:53.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.882+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb6e0045ad0 con 0x7fb6e00f6510 2024-01-27T23:32:53.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.882+0000 7fb6d1ffb700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb6cc02ff60 0x7fb6cc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:53.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.882+0000 7fb6d1ffb700 1 -- 172.21.15.194:0/3734685520 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb6d4041050 con 0x7fb6e00f6510 2024-01-27T23:32:53.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.882+0000 7fb6e4d50700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb6cc02ff60 0x7fb6cc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:53.878 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.883+0000 7fb6e4d50700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb6cc02ff60 0x7fb6cc032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb6dc0096a0 tx=0x7fb6dc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:53.881 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:53.887+0000 7fb6d1ffb700 1 -- 172.21.15.194:0/3734685520 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb6d40095f0 con 0x7fb6e00f6510 2024-01-27T23:32:54.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.095+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb6e00fbf60 con 0x7fb6e00f6510 2024-01-27T23:32:54.090 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.096+0000 7fb6d1ffb700 1 -- 172.21.15.194:0/3734685520 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb6d401db70 con 0x7fb6e00f6510 2024-01-27T23:32:54.090 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:54.090 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:54.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb6cc02ff60 msgr2=0x7fb6cc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:54.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb6cc02ff60 0x7fb6cc032420 secure :-1 s=READY pgs=164 cs=0 l=1 rev1=1 rx=0x7fb6dc0096a0 tx=0x7fb6dc006b40).stop 2024-01-27T23:32:54.092 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 msgr2=0x7fb6e0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e0106730 secure :-1 s=READY pgs=396 cs=0 l=1 rev1=1 rx=0x7fb6d4013510 tx=0x7fb6d400b040).stop 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 shutdown_connections 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb6cc02ff60 0x7fb6cc032420 unknown :-1 s=CLOSED pgs=164 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 --2- 172.21.15.194:0/3734685520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb6e00f6510 0x7fb6e0106730 unknown :-1 s=CLOSED pgs=396 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.097+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 >> 172.21.15.194:0/3734685520 conn(0x7fb6e00f1c70 msgr2=0x7fb6e00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.098+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 shutdown_connections 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:54.098+0000 7fb6e77b5700 1 -- 172.21.15.194:0/3734685520 wait complete. 2024-01-27T23:32:54.093 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:54.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:53.328797+0000 mgr.smithi121.hcppvz (mgr.14162) 236 : cluster [DBG] pgmap v222: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:54.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:54.097056+0000 mon.smithi121 (mon.0) 294 : audit [DBG] from='client.? 172.21.15.194:0/3734685520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:55.508 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:55.508 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:56.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.635+0000 7f1ee90b1700 1 -- 172.21.15.194:0/3626471995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 msgr2=0x7f1ee40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:56.631 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.635+0000 7f1ee90b1700 1 --2- 172.21.15.194:0/3626471995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee40f5830 secure :-1 s=READY pgs=397 cs=0 l=1 rev1=1 rx=0x7f1ecc009fb0 tx=0x7f1ecc009b30).stop 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.635+0000 7f1ee90b1700 1 -- 172.21.15.194:0/3626471995 shutdown_connections 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.635+0000 7f1ee90b1700 1 --2- 172.21.15.194:0/3626471995 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee40f5830 unknown :-1 s=CLOSED pgs=397 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.635+0000 7f1ee90b1700 1 -- 172.21.15.194:0/3626471995 >> 172.21.15.194:0/3626471995 conn(0x7f1ee40f0b70 msgr2=0x7f1ee40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.635+0000 7f1ee90b1700 1 -- 172.21.15.194:0/3626471995 shutdown_connections 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.635+0000 7f1ee90b1700 1 -- 172.21.15.194:0/3626471995 wait complete. 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.636+0000 7f1ee90b1700 1 Processor -- start 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.636+0000 7f1ee90b1700 1 -- start start 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.636+0000 7f1ee90b1700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee4109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.636+0000 7f1ee90b1700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f1ee4060240 con 0x7f1ee40f5410 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.636+0000 7f1ee2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee4109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.636+0000 7f1ee2d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee4109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:47672/0 (socket says 172.21.15.194:47672) 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.636+0000 7f1ee2d9d700 1 -- 172.21.15.194:0/1887981931 learned_addr learned my addr 172.21.15.194:0/1887981931 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.637+0000 7f1ee2d9d700 1 -- 172.21.15.194:0/1887981931 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f1ecc005040 con 0x7f1ee40f5410 2024-01-27T23:32:56.632 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.637+0000 7f1ee2d9d700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee4109430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f1ecc013510 tx=0x7f1ecc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:56.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.637+0000 7f1edb7fe700 1 -- 172.21.15.194:0/1887981931 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ecc009320 con 0x7f1ee40f5410 2024-01-27T23:32:56.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.637+0000 7f1edb7fe700 1 -- 172.21.15.194:0/1887981931 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f1ecc013f00 con 0x7f1ee40f5410 2024-01-27T23:32:56.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.637+0000 7f1edb7fe700 1 -- 172.21.15.194:0/1887981931 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f1ecc014d90 con 0x7f1ee40f5410 2024-01-27T23:32:56.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.637+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f1ee4060440 con 0x7f1ee40f5410 2024-01-27T23:32:56.633 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.637+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f1ee4109ae0 con 0x7f1ee40f5410 2024-01-27T23:32:56.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.638+0000 7f1edb7fe700 1 -- 172.21.15.194:0/1887981931 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f1ecc014420 con 0x7f1ee40f5410 2024-01-27T23:32:56.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.638+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f1ee4045ad0 con 0x7f1ee40f5410 2024-01-27T23:32:56.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.638+0000 7f1edb7fe700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1ed002ff60 0x7f1ed0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:56.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.638+0000 7f1edb7fe700 1 -- 172.21.15.194:0/1887981931 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f1ecc0410c0 con 0x7f1ee40f5410 2024-01-27T23:32:56.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.638+0000 7f1ee259c700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1ed002ff60 0x7f1ed0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:56.634 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.638+0000 7f1ee259c700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1ed002ff60 0x7f1ed0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f1ed40096a0 tx=0x7f1ed4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:56.637 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.643+0000 7f1edb7fe700 1 -- 172.21.15.194:0/1887981931 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f1ecc0095f0 con 0x7f1ee40f5410 2024-01-27T23:32:56.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:55.329079+0000 mgr.smithi121.hcppvz (mgr.14162) 237 : cluster [DBG] pgmap v223: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:56.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.848+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f1ee40fae60 con 0x7f1ee40f5410 2024-01-27T23:32:56.842 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.848+0000 7f1edb7fe700 1 -- 172.21.15.194:0/1887981931 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f1ecc01db70 con 0x7f1ee40f5410 2024-01-27T23:32:56.843 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:56.843 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:56.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1ed002ff60 msgr2=0x7f1ed0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:56.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1ed002ff60 0x7f1ed0032420 secure :-1 s=READY pgs=165 cs=0 l=1 rev1=1 rx=0x7f1ed40096a0 tx=0x7f1ed4006b40).stop 2024-01-27T23:32:56.844 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 msgr2=0x7f1ee4109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee4109430 secure :-1 s=READY pgs=398 cs=0 l=1 rev1=1 rx=0x7f1ecc013510 tx=0x7f1ecc00b040).stop 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 shutdown_connections 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f1ed002ff60 0x7f1ed0032420 unknown :-1 s=CLOSED pgs=165 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 --2- 172.21.15.194:0/1887981931 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f1ee40f5410 0x7f1ee4109430 unknown :-1 s=CLOSED pgs=398 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 >> 172.21.15.194:0/1887981931 conn(0x7f1ee40f0b70 msgr2=0x7f1ee40f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 shutdown_connections 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:56.850+0000 7f1ee90b1700 1 -- 172.21.15.194:0/1887981931 wait complete. 2024-01-27T23:32:56.845 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:32:57.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:57 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:56.849594+0000 mon.smithi121 (mon.0) 295 : audit [DBG] from='client.? 172.21.15.194:0/1887981931' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:32:58.221 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:32:58.221 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:32:58.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:32:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:57.329407+0000 mgr.smithi121.hcppvz (mgr.14162) 238 : cluster [DBG] pgmap v224: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:32:59.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.472+0000 7ff549408700 1 -- 172.21.15.194:0/4092702067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 msgr2=0x7ff5440f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:59.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.472+0000 7ff549408700 1 --2- 172.21.15.194:0/4092702067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440f5830 secure :-1 s=READY pgs=399 cs=0 l=1 rev1=1 rx=0x7ff538009fb0 tx=0x7ff538009b30).stop 2024-01-27T23:32:59.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.472+0000 7ff549408700 1 -- 172.21.15.194:0/4092702067 shutdown_connections 2024-01-27T23:32:59.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.472+0000 7ff549408700 1 --2- 172.21.15.194:0/4092702067 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440f5830 unknown :-1 s=CLOSED pgs=399 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:59.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.472+0000 7ff549408700 1 -- 172.21.15.194:0/4092702067 >> 172.21.15.194:0/4092702067 conn(0x7ff5440f0b70 msgr2=0x7ff5440f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:59.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.472+0000 7ff549408700 1 -- 172.21.15.194:0/4092702067 shutdown_connections 2024-01-27T23:32:59.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.472+0000 7ff549408700 1 -- 172.21.15.194:0/4092702067 wait complete. 2024-01-27T23:32:59.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.473+0000 7ff549408700 1 Processor -- start 2024-01-27T23:32:59.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.473+0000 7ff549408700 1 -- start start 2024-01-27T23:32:59.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.473+0000 7ff549408700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:59.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.473+0000 7ff549408700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7ff5440fb820 con 0x7ff5440f5410 2024-01-27T23:32:59.469 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.473+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.473+0000 7ff542ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:47690/0 (socket says 172.21.15.194:47690) 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.473+0000 7ff542ffd700 1 -- 172.21.15.194:0/2270521659 learned_addr learned my addr 172.21.15.194:0/2270521659 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.474+0000 7ff542ffd700 1 -- 172.21.15.194:0/2270521659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7ff538005040 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.474+0000 7ff542ffd700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440fd3a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff538013510 tx=0x7ff53800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.474+0000 7ff52b7fe700 1 -- 172.21.15.194:0/2270521659 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff538009320 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.474+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7ff5440fb9c0 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.474+0000 7ff52b7fe700 1 -- 172.21.15.194:0/2270521659 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7ff538013f00 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.474+0000 7ff52b7fe700 1 -- 172.21.15.194:0/2270521659 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7ff538014d90 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.474+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7ff5440fbe50 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.475+0000 7ff52b7fe700 1 -- 172.21.15.194:0/2270521659 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7ff5380095f0 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.475+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7ff544059100 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.475+0000 7ff52b7fe700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff52c02ff60 0x7ff52c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.475+0000 7ff52b7fe700 1 -- 172.21.15.194:0/2270521659 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7ff538040fc0 con 0x7ff5440f5410 2024-01-27T23:32:59.470 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.475+0000 7ff5427fc700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff52c02ff60 0x7ff52c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:32:59.471 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.476+0000 7ff5427fc700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff52c02ff60 0x7ff52c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff5340096a0 tx=0x7ff534006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:32:59.474 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.480+0000 7ff52b7fe700 1 -- 172.21.15.194:0/2270521659 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7ff538014420 con 0x7ff5440f5410 2024-01-27T23:32:59.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.684+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7ff544045ad0 con 0x7ff5440f5410 2024-01-27T23:32:59.680 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.685+0000 7ff52b7fe700 1 -- 172.21.15.194:0/2270521659 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7ff538010230 con 0x7ff5440f5410 2024-01-27T23:32:59.680 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:32:59.680 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:32:59.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff52c02ff60 msgr2=0x7ff52c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:59.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff52c02ff60 0x7ff52c032420 secure :-1 s=READY pgs=166 cs=0 l=1 rev1=1 rx=0x7ff5340096a0 tx=0x7ff534006b40).stop 2024-01-27T23:32:59.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 msgr2=0x7ff5440fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:32:59.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440fd3a0 secure :-1 s=READY pgs=400 cs=0 l=1 rev1=1 rx=0x7ff538013510 tx=0x7ff53800b040).stop 2024-01-27T23:32:59.683 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 shutdown_connections 2024-01-27T23:32:59.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7ff52c02ff60 0x7ff52c032420 unknown :-1 s=CLOSED pgs=166 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:59.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 --2- 172.21.15.194:0/2270521659 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7ff5440f5410 0x7ff5440fd3a0 unknown :-1 s=CLOSED pgs=400 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:32:59.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 >> 172.21.15.194:0/2270521659 conn(0x7ff5440f0b70 msgr2=0x7ff54405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:32:59.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 shutdown_connections 2024-01-27T23:32:59.684 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:32:59.687+0000 7ff549408700 1 -- 172.21.15.194:0/2270521659 wait complete. 2024-01-27T23:32:59.685 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:00.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:32:59.329791+0000 mgr.smithi121.hcppvz (mgr.14162) 239 : cluster [DBG] pgmap v225: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:00.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:32:59.686593+0000 mon.smithi121 (mon.0) 296 : audit [DBG] from='client.? 172.21.15.194:0/2270521659' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:01.071 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:01.071 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.192+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/894106703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 msgr2=0x7f0f140f6db0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.192+0000 7f0f1a9fd700 1 --2- 172.21.15.194:0/894106703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f140f6db0 secure :-1 s=READY pgs=401 cs=0 l=1 rev1=1 rx=0x7f0f08009fb0 tx=0x7f0f08009b30).stop 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.192+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/894106703 shutdown_connections 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.192+0000 7f0f1a9fd700 1 --2- 172.21.15.194:0/894106703 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f140f6db0 unknown :-1 s=CLOSED pgs=401 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.192+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/894106703 >> 172.21.15.194:0/894106703 conn(0x7f0f140f0260 msgr2=0x7f0f140f2680 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.192+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/894106703 shutdown_connections 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.192+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/894106703 wait complete. 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.193+0000 7f0f1a9fd700 1 Processor -- start 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.193+0000 7f0f1a9fd700 1 -- start start 2024-01-27T23:33:02.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.193+0000 7f0f1a9fd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f14106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:02.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.193+0000 7f0f1a9fd700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f0f14106c10 con 0x7f0f140f6990 2024-01-27T23:33:02.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.193+0000 7f0f13fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f14106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:02.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.193+0000 7f0f13fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f14106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:47698/0 (socket says 172.21.15.194:47698) 2024-01-27T23:33:02.189 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.193+0000 7f0f13fff700 1 -- 172.21.15.194:0/279683807 learned_addr learned my addr 172.21.15.194:0/279683807 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.194+0000 7f0f13fff700 1 -- 172.21.15.194:0/279683807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0f08005040 con 0x7f0f140f6990 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.194+0000 7f0f13fff700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f14106720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0f08013510 tx=0x7f0f0800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.194+0000 7f0f10ff9700 1 -- 172.21.15.194:0/279683807 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f08009320 con 0x7f0f140f6990 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.194+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f0f14106e10 con 0x7f0f140f6990 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.194+0000 7f0f10ff9700 1 -- 172.21.15.194:0/279683807 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0f08013f00 con 0x7f0f140f6990 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.194+0000 7f0f10ff9700 1 -- 172.21.15.194:0/279683807 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0f08014d90 con 0x7f0f140f6990 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.194+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f0f141072a0 con 0x7f0f140f6990 2024-01-27T23:33:02.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.195+0000 7f0f10ff9700 1 -- 172.21.15.194:0/279683807 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f0f080095f0 con 0x7f0f140f6990 2024-01-27T23:33:02.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.195+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f0f14045ad0 con 0x7f0f140f6990 2024-01-27T23:33:02.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.195+0000 7f0f10ff9700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0efc02ff60 0x7f0efc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:02.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.195+0000 7f0f10ff9700 1 -- 172.21.15.194:0/279683807 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0f08041080 con 0x7f0f140f6990 2024-01-27T23:33:02.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.195+0000 7f0f137fe700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0efc02ff60 0x7f0efc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:02.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.195+0000 7f0f137fe700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0efc02ff60 0x7f0efc032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0f040096a0 tx=0x7f0f04006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:02.194 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.200+0000 7f0f10ff9700 1 -- 172.21.15.194:0/279683807 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0f08014420 con 0x7f0f140f6990 2024-01-27T23:33:02.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.406+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f0f14108150 con 0x7f0f140f6990 2024-01-27T23:33:02.401 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.407+0000 7f0f10ff9700 1 -- 172.21.15.194:0/279683807 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0f0801d7f0 con 0x7f0f140f6990 2024-01-27T23:33:02.402 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:02.402 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:02.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0efc02ff60 msgr2=0x7f0efc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:02.403 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0efc02ff60 0x7f0efc032420 secure :-1 s=READY pgs=167 cs=0 l=1 rev1=1 rx=0x7f0f040096a0 tx=0x7f0f04006b40).stop 2024-01-27T23:33:02.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 msgr2=0x7f0f14106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:02.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f14106720 secure :-1 s=READY pgs=402 cs=0 l=1 rev1=1 rx=0x7f0f08013510 tx=0x7f0f0800b040).stop 2024-01-27T23:33:02.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 shutdown_connections 2024-01-27T23:33:02.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f0efc02ff60 0x7f0efc032420 unknown :-1 s=CLOSED pgs=167 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:02.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 --2- 172.21.15.194:0/279683807 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f0f140f6990 0x7f0f14106720 unknown :-1 s=CLOSED pgs=402 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:02.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 >> 172.21.15.194:0/279683807 conn(0x7f0f140f0260 msgr2=0x7f0f140f4eb0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:02.404 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 shutdown_connections 2024-01-27T23:33:02.405 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:02.409+0000 7f0f1a9fd700 1 -- 172.21.15.194:0/279683807 wait complete. 2024-01-27T23:33:02.405 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:02.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:01.330151+0000 mgr.smithi121.hcppvz (mgr.14162) 240 : cluster [DBG] pgmap v226: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:03.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:03 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:02.408204+0000 mon.smithi121 (mon.0) 297 : audit [DBG] from='client.? 172.21.15.194:0/279683807' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:03.793 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:03.793 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:04.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:03.330396+0000 mgr.smithi121.hcppvz (mgr.14162) 241 : cluster [DBG] pgmap v227: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:04.942 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.947+0000 7fa270683700 1 -- 172.21.15.194:0/2107723783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 msgr2=0x7fa2680f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.947+0000 7fa270683700 1 --2- 172.21.15.194:0/2107723783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa2680f5830 secure :-1 s=READY pgs=403 cs=0 l=1 rev1=1 rx=0x7fa258009fb0 tx=0x7fa258009b30).stop 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.947+0000 7fa270683700 1 -- 172.21.15.194:0/2107723783 shutdown_connections 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.947+0000 7fa270683700 1 --2- 172.21.15.194:0/2107723783 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa2680f5830 unknown :-1 s=CLOSED pgs=403 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.947+0000 7fa270683700 1 -- 172.21.15.194:0/2107723783 >> 172.21.15.194:0/2107723783 conn(0x7fa2680f0b70 msgr2=0x7fa2680f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.947+0000 7fa270683700 1 -- 172.21.15.194:0/2107723783 shutdown_connections 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.947+0000 7fa270683700 1 -- 172.21.15.194:0/2107723783 wait complete. 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.948+0000 7fa270683700 1 Processor -- start 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.948+0000 7fa270683700 1 -- start start 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.948+0000 7fa270683700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa268109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:04.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.948+0000 7fa270683700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa268060220 con 0x7fa2680f5410 2024-01-27T23:33:04.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.948+0000 7fa26e41f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa268109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:04.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.948+0000 7fa26e41f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa268109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:47716/0 (socket says 172.21.15.194:47716) 2024-01-27T23:33:04.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.948+0000 7fa26e41f700 1 -- 172.21.15.194:0/789126755 learned_addr learned my addr 172.21.15.194:0/789126755 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:04.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.949+0000 7fa26e41f700 1 -- 172.21.15.194:0/789126755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa258005040 con 0x7fa2680f5410 2024-01-27T23:33:04.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.949+0000 7fa26e41f700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa268109450 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa258013510 tx=0x7fa25800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:04.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.949+0000 7fa25effd700 1 -- 172.21.15.194:0/789126755 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa258009320 con 0x7fa2680f5410 2024-01-27T23:33:04.945 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.949+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa268060420 con 0x7fa2680f5410 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.949+0000 7fa25effd700 1 -- 172.21.15.194:0/789126755 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa258013f00 con 0x7fa2680f5410 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.949+0000 7fa25effd700 1 -- 172.21.15.194:0/789126755 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa258014d90 con 0x7fa2680f5410 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.949+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa268109b00 con 0x7fa2680f5410 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.950+0000 7fa25effd700 1 -- 172.21.15.194:0/789126755 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa2580263a0 con 0x7fa2680f5410 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.950+0000 7fa25effd700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa25402ff60 0x7fa254032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.950+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa268059100 con 0x7fa2680f5410 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.950+0000 7fa25effd700 1 -- 172.21.15.194:0/789126755 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa2580410b0 con 0x7fa2680f5410 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.950+0000 7fa26dc1e700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa25402ff60 0x7fa254032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:04.946 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.950+0000 7fa26dc1e700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa25402ff60 0x7fa254032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa2640096a0 tx=0x7fa264006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:04.949 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:04.955+0000 7fa25effd700 1 -- 172.21.15.194:0/789126755 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa258012920 con 0x7fa2680f5410 2024-01-27T23:33:05.158 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.163+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa268045ad0 con 0x7fa2680f5410 2024-01-27T23:33:05.159 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.164+0000 7fa25effd700 1 -- 172.21.15.194:0/789126755 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa258012330 con 0x7fa2680f5410 2024-01-27T23:33:05.159 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:05.159 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa25402ff60 msgr2=0x7fa254032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa25402ff60 0x7fa254032420 secure :-1 s=READY pgs=168 cs=0 l=1 rev1=1 rx=0x7fa2640096a0 tx=0x7fa264006b40).stop 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 msgr2=0x7fa268109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa268109450 secure :-1 s=READY pgs=404 cs=0 l=1 rev1=1 rx=0x7fa258013510 tx=0x7fa25800b040).stop 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 shutdown_connections 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa25402ff60 0x7fa254032420 unknown :-1 s=CLOSED pgs=168 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 --2- 172.21.15.194:0/789126755 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa2680f5410 0x7fa268109450 unknown :-1 s=CLOSED pgs=404 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 >> 172.21.15.194:0/789126755 conn(0x7fa2680f0b70 msgr2=0x7fa2680f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 shutdown_connections 2024-01-27T23:33:05.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:05.166+0000 7fa270683700 1 -- 172.21.15.194:0/789126755 wait complete. 2024-01-27T23:33:05.162 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:05.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:05 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:05.165567+0000 mon.smithi121 (mon.0) 298 : audit [DBG] from='client.? 172.21.15.194:0/789126755' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:06.550 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:06.551 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:06.591 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:05.330592+0000 mgr.smithi121.hcppvz (mgr.14162) 242 : cluster [DBG] pgmap v228: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.744+0000 7fd73090a700 1 -- 172.21.15.194:0/641791528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 msgr2=0x7fd72c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.744+0000 7fd73090a700 1 --2- 172.21.15.194:0/641791528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c0f5830 secure :-1 s=READY pgs=405 cs=0 l=1 rev1=1 rx=0x7fd720009fb0 tx=0x7fd720009b30).stop 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 -- 172.21.15.194:0/641791528 shutdown_connections 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 --2- 172.21.15.194:0/641791528 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c0f5830 unknown :-1 s=CLOSED pgs=405 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 -- 172.21.15.194:0/641791528 >> 172.21.15.194:0/641791528 conn(0x7fd72c0f0b70 msgr2=0x7fd72c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 -- 172.21.15.194:0/641791528 shutdown_connections 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 -- 172.21.15.194:0/641791528 wait complete. 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 Processor -- start 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 -- start start 2024-01-27T23:33:07.741 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.745+0000 7fd73090a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd72c106c10 con 0x7fd72c0f5410 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd72a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd72a59c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40240/0 (socket says 172.21.15.194:40240) 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd72a59c700 1 -- 172.21.15.194:0/366127557 learned_addr learned my addr 172.21.15.194:0/366127557 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd72a59c700 1 -- 172.21.15.194:0/366127557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd720005040 con 0x7fd72c0f5410 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd72a59c700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c106720 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fd720013510 tx=0x7fd72000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd71affd700 1 -- 172.21.15.194:0/366127557 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd720009320 con 0x7fd72c0f5410 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd72c106e10 con 0x7fd72c0f5410 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd71affd700 1 -- 172.21.15.194:0/366127557 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd720013f00 con 0x7fd72c0f5410 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd71affd700 1 -- 172.21.15.194:0/366127557 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd720014d90 con 0x7fd72c0f5410 2024-01-27T23:33:07.742 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.746+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd72c1072a0 con 0x7fd72c0f5410 2024-01-27T23:33:07.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.747+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd72c045ad0 con 0x7fd72c0f5410 2024-01-27T23:33:07.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.747+0000 7fd71affd700 1 -- 172.21.15.194:0/366127557 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd7200095f0 con 0x7fd72c0f5410 2024-01-27T23:33:07.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.747+0000 7fd71affd700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd71402ff60 0x7fd714032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:07.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.747+0000 7fd71affd700 1 -- 172.21.15.194:0/366127557 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd720040fc0 con 0x7fd72c0f5410 2024-01-27T23:33:07.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.747+0000 7fd729d9b700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd71402ff60 0x7fd714032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:07.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.748+0000 7fd729d9b700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd71402ff60 0x7fd714032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd71c0096a0 tx=0x7fd71c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:07.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.752+0000 7fd71affd700 1 -- 172.21.15.194:0/366127557 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd720014420 con 0x7fd72c0f5410 2024-01-27T23:33:07.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.957+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd72c1080f0 con 0x7fd72c0f5410 2024-01-27T23:33:07.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.958+0000 7fd71affd700 1 -- 172.21.15.194:0/366127557 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd720010230 con 0x7fd72c0f5410 2024-01-27T23:33:07.953 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:07.953 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd71402ff60 msgr2=0x7fd714032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd71402ff60 0x7fd714032420 secure :-1 s=READY pgs=169 cs=0 l=1 rev1=1 rx=0x7fd71c0096a0 tx=0x7fd71c006b40).stop 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 msgr2=0x7fd72c106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c106720 secure :-1 s=READY pgs=406 cs=0 l=1 rev1=1 rx=0x7fd720013510 tx=0x7fd72000b040).stop 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 shutdown_connections 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd71402ff60 0x7fd714032420 unknown :-1 s=CLOSED pgs=169 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 --2- 172.21.15.194:0/366127557 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd72c0f5410 0x7fd72c106720 unknown :-1 s=CLOSED pgs=406 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 >> 172.21.15.194:0/366127557 conn(0x7fd72c0f0b70 msgr2=0x7fd72c05fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 shutdown_connections 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:07.960+0000 7fd73090a700 1 -- 172.21.15.194:0/366127557 wait complete. 2024-01-27T23:33:07.955 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:08.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:07.330861+0000 mgr.smithi121.hcppvz (mgr.14162) 243 : cluster [DBG] pgmap v229: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:08.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:07.959590+0000 mon.smithi121 (mon.0) 299 : audit [DBG] from='client.? 172.21.15.194:0/366127557' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:09.348 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:09.348 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:10.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.534+0000 7f7348eeb700 1 -- 172.21.15.194:0/629099609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 msgr2=0x7f73440f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:10.530 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.534+0000 7f7348eeb700 1 --2- 172.21.15.194:0/629099609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f73440f6930 secure :-1 s=READY pgs=407 cs=0 l=1 rev1=1 rx=0x7f7338009fb0 tx=0x7f7338009b30).stop 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.534+0000 7f7348eeb700 1 -- 172.21.15.194:0/629099609 shutdown_connections 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.534+0000 7f7348eeb700 1 --2- 172.21.15.194:0/629099609 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f73440f6930 unknown :-1 s=CLOSED pgs=407 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.534+0000 7f7348eeb700 1 -- 172.21.15.194:0/629099609 >> 172.21.15.194:0/629099609 conn(0x7f73440f1c70 msgr2=0x7f73440f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.534+0000 7f7348eeb700 1 -- 172.21.15.194:0/629099609 shutdown_connections 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.534+0000 7f7348eeb700 1 -- 172.21.15.194:0/629099609 wait complete. 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.535+0000 7f7348eeb700 1 Processor -- start 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.535+0000 7f7348eeb700 1 -- start start 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.535+0000 7f7348eeb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f7344106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.535+0000 7f7348eeb700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7344106c20 con 0x7f73440f6510 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.535+0000 7f734259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f7344106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.535+0000 7f734259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f7344106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40262/0 (socket says 172.21.15.194:40262) 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.535+0000 7f734259c700 1 -- 172.21.15.194:0/2760084521 learned_addr learned my addr 172.21.15.194:0/2760084521 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.536+0000 7f734259c700 1 -- 172.21.15.194:0/2760084521 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7338005040 con 0x7f73440f6510 2024-01-27T23:33:10.531 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.536+0000 7f734259c700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f7344106730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f7338013510 tx=0x7f733800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:10.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.536+0000 7f7332ffd700 1 -- 172.21.15.194:0/2760084521 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7338009320 con 0x7f73440f6510 2024-01-27T23:33:10.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.536+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7344106e20 con 0x7f73440f6510 2024-01-27T23:33:10.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.536+0000 7f7332ffd700 1 -- 172.21.15.194:0/2760084521 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7338013f00 con 0x7f73440f6510 2024-01-27T23:33:10.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.536+0000 7f7332ffd700 1 -- 172.21.15.194:0/2760084521 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7338014d90 con 0x7f73440f6510 2024-01-27T23:33:10.532 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.536+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f73441072b0 con 0x7f73440f6510 2024-01-27T23:33:10.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.537+0000 7f7332ffd700 1 -- 172.21.15.194:0/2760084521 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f73380263a0 con 0x7f73440f6510 2024-01-27T23:33:10.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.537+0000 7f7332ffd700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f732c02ff60 0x7f732c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:10.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.537+0000 7f7332ffd700 1 -- 172.21.15.194:0/2760084521 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f73380410b0 con 0x7f73440f6510 2024-01-27T23:33:10.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.537+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7344045ad0 con 0x7f73440f6510 2024-01-27T23:33:10.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.537+0000 7f7341d9b700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f732c02ff60 0x7f732c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:10.533 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.538+0000 7f7341d9b700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f732c02ff60 0x7f732c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f73340096a0 tx=0x7f7334006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:10.536 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.542+0000 7f7332ffd700 1 -- 172.21.15.194:0/2760084521 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f7338012920 con 0x7f73440f6510 2024-01-27T23:33:10.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.748+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f7344107fd0 con 0x7f73440f6510 2024-01-27T23:33:10.743 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.749+0000 7f7332ffd700 1 -- 172.21.15.194:0/2760084521 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f733802b020 con 0x7f73440f6510 2024-01-27T23:33:10.744 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:10.744 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:10.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.750+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f732c02ff60 msgr2=0x7f732c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:10.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.750+0000 7f7348eeb700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f732c02ff60 0x7f732c032420 secure :-1 s=READY pgs=170 cs=0 l=1 rev1=1 rx=0x7f73340096a0 tx=0x7f7334006b40).stop 2024-01-27T23:33:10.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.750+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 msgr2=0x7f7344106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:10.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.750+0000 7f7348eeb700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f7344106730 secure :-1 s=READY pgs=408 cs=0 l=1 rev1=1 rx=0x7f7338013510 tx=0x7f733800b040).stop 2024-01-27T23:33:10.745 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.751+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 shutdown_connections 2024-01-27T23:33:10.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.751+0000 7f7348eeb700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f732c02ff60 0x7f732c032420 unknown :-1 s=CLOSED pgs=170 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:10.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.751+0000 7f7348eeb700 1 --2- 172.21.15.194:0/2760084521 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f73440f6510 0x7f7344106730 unknown :-1 s=CLOSED pgs=408 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:10.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.751+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 >> 172.21.15.194:0/2760084521 conn(0x7f73440f1c70 msgr2=0x7f73440f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:10.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.751+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 shutdown_connections 2024-01-27T23:33:10.746 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:10.751+0000 7f7348eeb700 1 -- 172.21.15.194:0/2760084521 wait complete. 2024-01-27T23:33:10.746 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:10.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:09.331142+0000 mgr.smithi121.hcppvz (mgr.14162) 244 : cluster [DBG] pgmap v230: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:11.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:11 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:10.750105+0000 mon.smithi121 (mon.0) 300 : audit [DBG] from='client.? 172.21.15.194:0/2760084521' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:12.130 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:12.131 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:12.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:11.331476+0000 mgr.smithi121.hcppvz (mgr.14162) 245 : cluster [DBG] pgmap v231: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:13.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/3372007052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 msgr2=0x7f9b780f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:13.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 --2- 172.21.15.194:0/3372007052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b780f66c0 secure :-1 s=READY pgs=409 cs=0 l=1 rev1=1 rx=0x7f9b6c009fb0 tx=0x7f9b6c009b30).stop 2024-01-27T23:33:13.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/3372007052 shutdown_connections 2024-01-27T23:33:13.273 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 --2- 172.21.15.194:0/3372007052 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b780f66c0 unknown :-1 s=CLOSED pgs=409 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:13.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/3372007052 >> 172.21.15.194:0/3372007052 conn(0x7f9b780f1aa0 msgr2=0x7f9b780f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:13.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/3372007052 shutdown_connections 2024-01-27T23:33:13.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/3372007052 wait complete. 2024-01-27T23:33:13.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 Processor -- start 2024-01-27T23:33:13.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.278+0000 7f9b7dfe4700 1 -- start start 2024-01-27T23:33:13.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.279+0000 7f9b7dfe4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b781064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:13.274 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.279+0000 7f9b7dfe4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9b781069a0 con 0x7f9b780f62a0 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.279+0000 7f9b7cfe2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b781064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.279+0000 7f9b7cfe2700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b781064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:40282/0 (socket says 172.21.15.194:40282) 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.279+0000 7f9b7cfe2700 1 -- 172.21.15.194:0/1891286725 learned_addr learned my addr 172.21.15.194:0/1891286725 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.279+0000 7f9b7cfe2700 1 -- 172.21.15.194:0/1891286725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9b6c005040 con 0x7f9b780f62a0 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.280+0000 7f9b7cfe2700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b781064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f9b6c013510 tx=0x7f9b6c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.280+0000 7f9b757fa700 1 -- 172.21.15.194:0/1891286725 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b6c009320 con 0x7f9b780f62a0 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.280+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9b78106ba0 con 0x7f9b780f62a0 2024-01-27T23:33:13.276 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.280+0000 7f9b757fa700 1 -- 172.21.15.194:0/1891286725 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9b6c013f00 con 0x7f9b780f62a0 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.280+0000 7f9b757fa700 1 -- 172.21.15.194:0/1891286725 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9b6c014dc0 con 0x7f9b780f62a0 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.280+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9b78107030 con 0x7f9b780f62a0 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.280+0000 7f9b757fa700 1 -- 172.21.15.194:0/1891286725 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9b6c01d3d0 con 0x7f9b780f62a0 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.281+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9b78045ad0 con 0x7f9b780f62a0 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.281+0000 7f9b757fa700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b6802ff60 0x7f9b68032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.281+0000 7f9b757fa700 1 -- 172.21.15.194:0/1891286725 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9b6c041320 con 0x7f9b780f62a0 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.281+0000 7f9b77fff700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b6802ff60 0x7f9b68032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:13.277 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.281+0000 7f9b77fff700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b6802ff60 0x7f9b68032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f9b640096a0 tx=0x7f9b64006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:13.280 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.285+0000 7f9b757fa700 1 -- 172.21.15.194:0/1891286725 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9b6c01cb50 con 0x7f9b780f62a0 2024-01-27T23:33:13.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.490+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9b78107ee0 con 0x7f9b780f62a0 2024-01-27T23:33:13.485 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.491+0000 7f9b757fa700 1 -- 172.21.15.194:0/1891286725 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9b6c01c970 con 0x7f9b780f62a0 2024-01-27T23:33:13.486 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:13.486 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:13.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b6802ff60 msgr2=0x7f9b68032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:13.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b6802ff60 0x7f9b68032420 secure :-1 s=READY pgs=171 cs=0 l=1 rev1=1 rx=0x7f9b640096a0 tx=0x7f9b64006b40).stop 2024-01-27T23:33:13.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 msgr2=0x7f9b781064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:13.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b781064b0 secure :-1 s=READY pgs=410 cs=0 l=1 rev1=1 rx=0x7f9b6c013510 tx=0x7f9b6c00b040).stop 2024-01-27T23:33:13.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 shutdown_connections 2024-01-27T23:33:13.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9b6802ff60 0x7f9b68032420 unknown :-1 s=CLOSED pgs=171 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:13.487 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 --2- 172.21.15.194:0/1891286725 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9b780f62a0 0x7f9b781064b0 unknown :-1 s=CLOSED pgs=410 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:13.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 >> 172.21.15.194:0/1891286725 conn(0x7f9b780f1aa0 msgr2=0x7f9b780f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:13.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 shutdown_connections 2024-01-27T23:33:13.488 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:13.492+0000 7f9b7dfe4700 1 -- 172.21.15.194:0/1891286725 wait complete. 2024-01-27T23:33:13.488 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:14.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:13.331757+0000 mgr.smithi121.hcppvz (mgr.14162) 246 : cluster [DBG] pgmap v232: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:14.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:13.492159+0000 mon.smithi121 (mon.0) 301 : audit [DBG] from='client.? 172.21.15.194:0/1891286725' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:14.864 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:14.864 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.994+0000 7f802890d700 1 -- 172.21.15.194:0/782572580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 msgr2=0x7f80240f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.994+0000 7f802890d700 1 --2- 172.21.15.194:0/782572580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80240f55c0 secure :-1 s=READY pgs=411 cs=0 l=1 rev1=1 rx=0x7f8018009fb0 tx=0x7f8018009b30).stop 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.994+0000 7f802890d700 1 -- 172.21.15.194:0/782572580 shutdown_connections 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.994+0000 7f802890d700 1 --2- 172.21.15.194:0/782572580 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80240f55c0 unknown :-1 s=CLOSED pgs=411 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.994+0000 7f802890d700 1 -- 172.21.15.194:0/782572580 >> 172.21.15.194:0/782572580 conn(0x7f80240f09a0 msgr2=0x7f80240f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.994+0000 7f802890d700 1 -- 172.21.15.194:0/782572580 shutdown_connections 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.994+0000 7f802890d700 1 -- 172.21.15.194:0/782572580 wait complete. 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.995+0000 7f802890d700 1 Processor -- start 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.995+0000 7f802890d700 1 -- start start 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.995+0000 7f802890d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80241091b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:15.990 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.995+0000 7f802890d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f8024060240 con 0x7f80240f51a0 2024-01-27T23:33:15.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.995+0000 7f8022d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80241091b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:15.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.995+0000 7f8022d9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80241091b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52554/0 (socket says 172.21.15.194:52554) 2024-01-27T23:33:15.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.995+0000 7f8022d9d700 1 -- 172.21.15.194:0/4167595261 learned_addr learned my addr 172.21.15.194:0/4167595261 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:15.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.996+0000 7f8022d9d700 1 -- 172.21.15.194:0/4167595261 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f8018005040 con 0x7f80240f51a0 2024-01-27T23:33:15.992 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.996+0000 7f8022d9d700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80241091b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f8018013510 tx=0x7f801800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.996+0000 7f800b7fe700 1 -- 172.21.15.194:0/4167595261 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8018009320 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.996+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f8024060440 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.996+0000 7f800b7fe700 1 -- 172.21.15.194:0/4167595261 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f8018013f00 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.996+0000 7f800b7fe700 1 -- 172.21.15.194:0/4167595261 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f8018014dc0 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.996+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f8024109860 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.997+0000 7f800b7fe700 1 -- 172.21.15.194:0/4167595261 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f801801d3d0 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.997+0000 7f800b7fe700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f800c02ff60 0x7f800c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.997+0000 7f800b7fe700 1 -- 172.21.15.194:0/4167595261 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f8018040ff0 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.997+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f8024045ad0 con 0x7f80240f51a0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.997+0000 7f802259c700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f800c02ff60 0x7f800c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:15.993 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:15.998+0000 7f802259c700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f800c02ff60 0x7f800c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f80140096a0 tx=0x7f8014006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:15.996 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.002+0000 7f800b7fe700 1 -- 172.21.15.194:0/4167595261 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f8018012900 con 0x7f80240f51a0 2024-01-27T23:33:16.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.208+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f8024109f40 con 0x7f80240f51a0 2024-01-27T23:33:16.203 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.209+0000 7f800b7fe700 1 -- 172.21.15.194:0/4167595261 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f801802b020 con 0x7f80240f51a0 2024-01-27T23:33:16.204 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:16.204 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:16.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.210+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f800c02ff60 msgr2=0x7f800c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:16.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.210+0000 7f802890d700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f800c02ff60 0x7f800c032420 secure :-1 s=READY pgs=172 cs=0 l=1 rev1=1 rx=0x7f80140096a0 tx=0x7f8014006b40).stop 2024-01-27T23:33:16.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.210+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 msgr2=0x7f80241091b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:16.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.210+0000 7f802890d700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80241091b0 secure :-1 s=READY pgs=412 cs=0 l=1 rev1=1 rx=0x7f8018013510 tx=0x7f801800b040).stop 2024-01-27T23:33:16.206 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.211+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 shutdown_connections 2024-01-27T23:33:16.207 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.211+0000 7f802890d700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f800c02ff60 0x7f800c032420 unknown :-1 s=CLOSED pgs=172 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:16.207 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.211+0000 7f802890d700 1 --2- 172.21.15.194:0/4167595261 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f80240f51a0 0x7f80241091b0 unknown :-1 s=CLOSED pgs=412 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:16.207 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.211+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 >> 172.21.15.194:0/4167595261 conn(0x7f80240f09a0 msgr2=0x7f80240f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:16.207 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.211+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 shutdown_connections 2024-01-27T23:33:16.207 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:16.211+0000 7f802890d700 1 -- 172.21.15.194:0/4167595261 wait complete. 2024-01-27T23:33:16.208 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:16.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:15.332075+0000 mgr.smithi121.hcppvz (mgr.14162) 247 : cluster [DBG] pgmap v233: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:16.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:16 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:16.210259+0000 mon.smithi121 (mon.0) 302 : audit [DBG] from='client.? 172.21.15.194:0/4167595261' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:17.621 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:17.621 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:18.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:17.332481+0000 mgr.smithi121.hcppvz (mgr.14162) 248 : cluster [DBG] pgmap v234: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:18.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:17.400489+0000 mon.smithi121 (mon.0) 303 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/mirror_snapshot_schedule"}]: dispatch 2024-01-27T23:33:18.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:18 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:17.407426+0000 mon.smithi121 (mon.0) 304 : audit [INF] from='mgr.14162 172.21.15.121:0/2734976596' entity='mgr.smithi121.hcppvz' cmd=[{"prefix":"config rm","who":"mgr","name":"mgr/rbd_support/smithi121.hcppvz/trash_purge_schedule"}]: dispatch 2024-01-27T23:33:18.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 -- 172.21.15.194:0/896458378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 msgr2=0x7f712c0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:18.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 --2- 172.21.15.194:0/896458378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c0f5830 secure :-1 s=READY pgs=413 cs=0 l=1 rev1=1 rx=0x7f7128009fb0 tx=0x7f7128009b30).stop 2024-01-27T23:33:18.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 -- 172.21.15.194:0/896458378 shutdown_connections 2024-01-27T23:33:18.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 --2- 172.21.15.194:0/896458378 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c0f5830 unknown :-1 s=CLOSED pgs=413 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 -- 172.21.15.194:0/896458378 >> 172.21.15.194:0/896458378 conn(0x7f712c0f0b70 msgr2=0x7f712c0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 -- 172.21.15.194:0/896458378 shutdown_connections 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 -- 172.21.15.194:0/896458378 wait complete. 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.919+0000 7f71333c4700 1 Processor -- start 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f71333c4700 1 -- start start 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f71333c4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c10b850 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f71333c4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f712c107240 con 0x7f712c0f5410 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f7131160700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c10b850 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f7131160700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c10b850 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52572/0 (socket says 172.21.15.194:52572) 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f7131160700 1 -- 172.21.15.194:0/3694225291 learned_addr learned my addr 172.21.15.194:0/3694225291 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f7131160700 1 -- 172.21.15.194:0/3694225291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f7128005040 con 0x7f712c0f5410 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f7131160700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c10b850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f7128013510 tx=0x7f712800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:18.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.920+0000 7f7121ffb700 1 -- 172.21.15.194:0/3694225291 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7128009320 con 0x7f712c0f5410 2024-01-27T23:33:18.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.921+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f712c060220 con 0x7f712c0f5410 2024-01-27T23:33:18.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.921+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f712c10bd40 con 0x7f712c0f5410 2024-01-27T23:33:18.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.921+0000 7f7121ffb700 1 -- 172.21.15.194:0/3694225291 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f7128013f00 con 0x7f712c0f5410 2024-01-27T23:33:18.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.921+0000 7f7121ffb700 1 -- 172.21.15.194:0/3694225291 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f7128014d60 con 0x7f712c0f5410 2024-01-27T23:33:18.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.921+0000 7f7121ffb700 1 -- 172.21.15.194:0/3694225291 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f71280263a0 con 0x7f712c0f5410 2024-01-27T23:33:18.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.922+0000 7f7121ffb700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f711802ff60 0x7f7118032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:18.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.922+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f712c045ad0 con 0x7f712c0f5410 2024-01-27T23:33:18.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.922+0000 7f7121ffb700 1 -- 172.21.15.194:0/3694225291 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f71280412f0 con 0x7f712c0f5410 2024-01-27T23:33:18.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.922+0000 7f713095f700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f711802ff60 0x7f7118032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:18.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.922+0000 7f713095f700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f711802ff60 0x7f7118032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f711c0096a0 tx=0x7f711c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:18.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:18.927+0000 7f7121ffb700 1 -- 172.21.15.194:0/3694225291 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f712801d380 con 0x7f712c0f5410 2024-01-27T23:33:19.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.132+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f712c10c500 con 0x7f712c0f5410 2024-01-27T23:33:19.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.132+0000 7f7121ffb700 1 -- 172.21.15.194:0/3694225291 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f712802b070 con 0x7f712c0f5410 2024-01-27T23:33:19.127 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:19.127 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:19.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f711802ff60 msgr2=0x7f7118032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:19.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f711802ff60 0x7f7118032420 secure :-1 s=READY pgs=173 cs=0 l=1 rev1=1 rx=0x7f711c0096a0 tx=0x7f711c006b40).stop 2024-01-27T23:33:19.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 msgr2=0x7f712c10b850 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:19.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c10b850 secure :-1 s=READY pgs=414 cs=0 l=1 rev1=1 rx=0x7f7128013510 tx=0x7f712800b040).stop 2024-01-27T23:33:19.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 shutdown_connections 2024-01-27T23:33:19.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f711802ff60 0x7f7118032420 unknown :-1 s=CLOSED pgs=173 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:19.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 --2- 172.21.15.194:0/3694225291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f712c0f5410 0x7f712c10b850 unknown :-1 s=CLOSED pgs=414 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:19.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 >> 172.21.15.194:0/3694225291 conn(0x7f712c0f0b70 msgr2=0x7f712c0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:19.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 shutdown_connections 2024-01-27T23:33:19.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:19.134+0000 7f71333c4700 1 -- 172.21.15.194:0/3694225291 wait complete. 2024-01-27T23:33:19.130 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:19.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:19 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:19.133978+0000 mon.smithi121 (mon.0) 305 : audit [DBG] from='client.? 172.21.15.194:0/3694225291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:20.530 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:20.530 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:20.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:20 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:19.332799+0000 mgr.smithi121.hcppvz (mgr.14162) 249 : cluster [DBG] pgmap v235: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.601+0000 7fd6fe534700 1 -- 172.21.15.194:0/1563311803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 msgr2=0x7fd6f80f9190 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.601+0000 7fd6fe534700 1 --2- 172.21.15.194:0/1563311803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f80f9190 secure :-1 s=READY pgs=415 cs=0 l=1 rev1=1 rx=0x7fd6e8009fb0 tx=0x7fd6e8009b30).stop 2024-01-27T23:33:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.601+0000 7fd6fe534700 1 -- 172.21.15.194:0/1563311803 shutdown_connections 2024-01-27T23:33:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.601+0000 7fd6fe534700 1 --2- 172.21.15.194:0/1563311803 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f80f9190 unknown :-1 s=CLOSED pgs=415 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:21.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.601+0000 7fd6fe534700 1 -- 172.21.15.194:0/1563311803 >> 172.21.15.194:0/1563311803 conn(0x7fd6f80f0b70 msgr2=0x7fd6f80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.601+0000 7fd6fe534700 1 -- 172.21.15.194:0/1563311803 shutdown_connections 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.601+0000 7fd6fe534700 1 -- 172.21.15.194:0/1563311803 wait complete. 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.602+0000 7fd6fe534700 1 Processor -- start 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.602+0000 7fd6fe534700 1 -- start start 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.602+0000 7fd6fe534700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f81066e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.602+0000 7fd6fe534700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fd6f8106bd0 con 0x7fd6f80601e0 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.602+0000 7fd6f7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f81066e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.602+0000 7fd6f7fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f81066e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52586/0 (socket says 172.21.15.194:52586) 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.602+0000 7fd6f7fff700 1 -- 172.21.15.194:0/2712343753 learned_addr learned my addr 172.21.15.194:0/2712343753 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.603+0000 7fd6f7fff700 1 -- 172.21.15.194:0/2712343753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fd6e8005040 con 0x7fd6f80601e0 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.603+0000 7fd6f7fff700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f81066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd6e8004dd0 tx=0x7fd6e8009b30).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.603+0000 7fd6f4ff9700 1 -- 172.21.15.194:0/2712343753 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6e8018070 con 0x7fd6f80601e0 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.603+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fd6f8106dd0 con 0x7fd6f80601e0 2024-01-27T23:33:21.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.603+0000 7fd6f4ff9700 1 -- 172.21.15.194:0/2712343753 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fd6e8013410 con 0x7fd6f80601e0 2024-01-27T23:33:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.603+0000 7fd6f4ff9700 1 -- 172.21.15.194:0/2712343753 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fd6e801c3c0 con 0x7fd6f80601e0 2024-01-27T23:33:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.603+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fd6f8107260 con 0x7fd6f80601e0 2024-01-27T23:33:21.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.604+0000 7fd6f4ff9700 1 -- 172.21.15.194:0/2712343753 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fd6e801c630 con 0x7fd6f80601e0 2024-01-27T23:33:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.604+0000 7fd6f4ff9700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd6e002ffb0 0x7fd6e0032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.604+0000 7fd6f4ff9700 1 -- 172.21.15.194:0/2712343753 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fd6e8041480 con 0x7fd6f80601e0 2024-01-27T23:33:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.604+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fd6f8045ad0 con 0x7fd6f80601e0 2024-01-27T23:33:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.604+0000 7fd6f77fe700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd6e002ffb0 0x7fd6e0032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:21.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.605+0000 7fd6f77fe700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd6e002ffb0 0x7fd6e0032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd6ec0096a0 tx=0x7fd6ec006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:21.603 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.609+0000 7fd6f4ff9700 1 -- 172.21.15.194:0/2712343753 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fd6e801ca30 con 0x7fd6f80601e0 2024-01-27T23:33:21.808 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.813+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fd6f8107ee0 con 0x7fd6f80601e0 2024-01-27T23:33:21.809 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.814+0000 7fd6f4ff9700 1 -- 172.21.15.194:0/2712343753 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fd6e801b9c0 con 0x7fd6f80601e0 2024-01-27T23:33:21.809 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:21.809 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:21.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd6e002ffb0 msgr2=0x7fd6e0032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:21.810 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd6e002ffb0 0x7fd6e0032470 secure :-1 s=READY pgs=174 cs=0 l=1 rev1=1 rx=0x7fd6ec0096a0 tx=0x7fd6ec006b40).stop 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 msgr2=0x7fd6f81066e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f81066e0 secure :-1 s=READY pgs=416 cs=0 l=1 rev1=1 rx=0x7fd6e8004dd0 tx=0x7fd6e8009b30).stop 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 shutdown_connections 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fd6e002ffb0 0x7fd6e0032470 unknown :-1 s=CLOSED pgs=174 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 --2- 172.21.15.194:0/2712343753 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fd6f80601e0 0x7fd6f81066e0 unknown :-1 s=CLOSED pgs=416 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 >> 172.21.15.194:0/2712343753 conn(0x7fd6f80f0b70 msgr2=0x7fd6f80f1670 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 shutdown_connections 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:21.816+0000 7fd6fe534700 1 -- 172.21.15.194:0/2712343753 wait complete. 2024-01-27T23:33:21.811 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:22.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:21.333061+0000 mgr.smithi121.hcppvz (mgr.14162) 250 : cluster [DBG] pgmap v236: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:22.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:22 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:21.815351+0000 mon.smithi121 (mon.0) 306 : audit [DBG] from='client.? 172.21.15.194:0/2712343753' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:23.191 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:23.191 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 -- 172.21.15.194:0/3571047946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 msgr2=0x7f57d80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 --2- 172.21.15.194:0/3571047946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d80f5830 secure :-1 s=READY pgs=417 cs=0 l=1 rev1=1 rx=0x7f57cc009fb0 tx=0x7f57cc009b30).stop 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 -- 172.21.15.194:0/3571047946 shutdown_connections 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 --2- 172.21.15.194:0/3571047946 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d80f5830 unknown :-1 s=CLOSED pgs=417 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 -- 172.21.15.194:0/3571047946 >> 172.21.15.194:0/3571047946 conn(0x7f57d80f0b70 msgr2=0x7f57d80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 -- 172.21.15.194:0/3571047946 shutdown_connections 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 -- 172.21.15.194:0/3571047946 wait complete. 2024-01-27T23:33:24.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 Processor -- start 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 -- start start 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d8109450 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.387+0000 7f57e0488700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f57d8060220 con 0x7f57d80f5410 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57de224700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d8109450 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57de224700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d8109450 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52602/0 (socket says 172.21.15.194:52602) 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57de224700 1 -- 172.21.15.194:0/1597680069 learned_addr learned my addr 172.21.15.194:0/1597680069 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57de224700 1 -- 172.21.15.194:0/1597680069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f57cc005040 con 0x7f57d80f5410 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57de224700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d8109450 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f57cc014000 tx=0x7f57cc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57caffd700 1 -- 172.21.15.194:0/1597680069 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57cc0095f0 con 0x7f57d80f5410 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f57d8060420 con 0x7f57d80f5410 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57caffd700 1 -- 172.21.15.194:0/1597680069 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f57cc007e70 con 0x7f57d80f5410 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57caffd700 1 -- 172.21.15.194:0/1597680069 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f57cc014ea0 con 0x7f57d80f5410 2024-01-27T23:33:24.384 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.388+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f57d8109b00 con 0x7f57d80f5410 2024-01-27T23:33:24.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.389+0000 7f57caffd700 1 -- 172.21.15.194:0/1597680069 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f57cc025070 con 0x7f57d80f5410 2024-01-27T23:33:24.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.389+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f57d8059100 con 0x7f57d80f5410 2024-01-27T23:33:24.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.389+0000 7f57caffd700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57c402ff60 0x7f57c4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:24.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.389+0000 7f57caffd700 1 -- 172.21.15.194:0/1597680069 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f57cc040940 con 0x7f57d80f5410 2024-01-27T23:33:24.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.389+0000 7f57dda23700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57c402ff60 0x7f57c4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:24.386 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.390+0000 7f57dda23700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57c402ff60 0x7f57c4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f57d40096a0 tx=0x7f57d4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:24.388 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.394+0000 7f57caffd700 1 -- 172.21.15.194:0/1597680069 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f57cc0079c0 con 0x7f57d80f5410 2024-01-27T23:33:24.598 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.603+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f57d8045ad0 con 0x7f57d80f5410 2024-01-27T23:33:24.599 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.604+0000 7f57caffd700 1 -- 172.21.15.194:0/1597680069 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f57cc00f300 con 0x7f57d80f5410 2024-01-27T23:33:24.599 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:24.599 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:24.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57c402ff60 msgr2=0x7f57c4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:24.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57c402ff60 0x7f57c4032420 secure :-1 s=READY pgs=175 cs=0 l=1 rev1=1 rx=0x7f57d40096a0 tx=0x7f57d4006b40).stop 2024-01-27T23:33:24.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 msgr2=0x7f57d8109450 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:24.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d8109450 secure :-1 s=READY pgs=418 cs=0 l=1 rev1=1 rx=0x7f57cc014000 tx=0x7f57cc00b040).stop 2024-01-27T23:33:24.600 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 shutdown_connections 2024-01-27T23:33:24.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f57c402ff60 0x7f57c4032420 unknown :-1 s=CLOSED pgs=175 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:24.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 --2- 172.21.15.194:0/1597680069 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f57d80f5410 0x7f57d8109450 unknown :-1 s=CLOSED pgs=418 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:24.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 >> 172.21.15.194:0/1597680069 conn(0x7f57d80f0b70 msgr2=0x7f57d80f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:24.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 shutdown_connections 2024-01-27T23:33:24.601 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:24.605+0000 7f57e0488700 1 -- 172.21.15.194:0/1597680069 wait complete. 2024-01-27T23:33:24.601 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:24.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:24 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:23.333446+0000 mgr.smithi121.hcppvz (mgr.14162) 251 : cluster [DBG] pgmap v237: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:25.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:25 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:24.604972+0000 mon.smithi121 (mon.0) 307 : audit [DBG] from='client.? 172.21.15.194:0/1597680069' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:25.987 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:25.987 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:26.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:26 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:25.333737+0000 mgr.smithi121.hcppvz (mgr.14162) 252 : cluster [DBG] pgmap v238: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:27.182 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.185+0000 7f0481b67700 1 -- 172.21.15.194:0/3743357856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 msgr2=0x7f047c0f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.185+0000 7f0481b67700 1 --2- 172.21.15.194:0/3743357856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c0f6930 secure :-1 s=READY pgs=419 cs=0 l=1 rev1=1 rx=0x7f0470009fb0 tx=0x7f0470009b30).stop 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.185+0000 7f0481b67700 1 -- 172.21.15.194:0/3743357856 shutdown_connections 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.185+0000 7f0481b67700 1 --2- 172.21.15.194:0/3743357856 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c0f6930 unknown :-1 s=CLOSED pgs=419 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.185+0000 7f0481b67700 1 -- 172.21.15.194:0/3743357856 >> 172.21.15.194:0/3743357856 conn(0x7f047c0f1c70 msgr2=0x7f047c0f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.185+0000 7f0481b67700 1 -- 172.21.15.194:0/3743357856 shutdown_connections 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.185+0000 7f0481b67700 1 -- 172.21.15.194:0/3743357856 wait complete. 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f0481b67700 1 Processor -- start 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f0481b67700 1 -- start start 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f0481b67700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:27.183 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f0481b67700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f047c106c20 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f047b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f047b7fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55476/0 (socket says 172.21.15.194:55476) 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f047b7fe700 1 -- 172.21.15.194:0/3307358817 learned_addr learned my addr 172.21.15.194:0/3307358817 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f047b7fe700 1 -- 172.21.15.194:0/3307358817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0470005040 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.186+0000 7f047b7fe700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c106730 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0470013510 tx=0x7f047000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.187+0000 7f0463fff700 1 -- 172.21.15.194:0/3307358817 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0470009320 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.187+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f047c106e20 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.187+0000 7f0463fff700 1 -- 172.21.15.194:0/3307358817 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0470013f00 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.187+0000 7f0463fff700 1 -- 172.21.15.194:0/3307358817 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0470014d90 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.187+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f047c1072b0 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.188+0000 7f0463fff700 1 -- 172.21.15.194:0/3307358817 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f04700095f0 con 0x7f047c0f6510 2024-01-27T23:33:27.184 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.188+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f047c045ad0 con 0x7f047c0f6510 2024-01-27T23:33:27.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.188+0000 7f0463fff700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f046402ff60 0x7f0464032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:27.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.188+0000 7f0463fff700 1 -- 172.21.15.194:0/3307358817 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f0470040fc0 con 0x7f047c0f6510 2024-01-27T23:33:27.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.188+0000 7f047affd700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f046402ff60 0x7f0464032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:27.185 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.188+0000 7f047affd700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f046402ff60 0x7f0464032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f046c0096a0 tx=0x7f046c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:27.187 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.192+0000 7f0463fff700 1 -- 172.21.15.194:0/3307358817 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f0470014420 con 0x7f047c0f6510 2024-01-27T23:33:27.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.396+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f047c0fbf60 con 0x7f047c0f6510 2024-01-27T23:33:27.391 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.397+0000 7f0463fff700 1 -- 172.21.15.194:0/3307358817 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0470010230 con 0x7f047c0f6510 2024-01-27T23:33:27.392 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:27.392 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:27.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f046402ff60 msgr2=0x7f0464032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:27.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f046402ff60 0x7f0464032420 secure :-1 s=READY pgs=176 cs=0 l=1 rev1=1 rx=0x7f046c0096a0 tx=0x7f046c006b40).stop 2024-01-27T23:33:27.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 msgr2=0x7f047c106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:27.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c106730 secure :-1 s=READY pgs=420 cs=0 l=1 rev1=1 rx=0x7f0470013510 tx=0x7f047000b040).stop 2024-01-27T23:33:27.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 shutdown_connections 2024-01-27T23:33:27.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f046402ff60 0x7f0464032420 unknown :-1 s=CLOSED pgs=176 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:27.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 --2- 172.21.15.194:0/3307358817 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f047c0f6510 0x7f047c106730 unknown :-1 s=CLOSED pgs=420 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:27.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 >> 172.21.15.194:0/3307358817 conn(0x7f047c0f1c70 msgr2=0x7f047c0f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:27.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 shutdown_connections 2024-01-27T23:33:27.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:27.398+0000 7f0481b67700 1 -- 172.21.15.194:0/3307358817 wait complete. 2024-01-27T23:33:27.395 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:28.820 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:28.821 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:28.841 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:27.334076+0000 mgr.smithi121.hcppvz (mgr.14162) 253 : cluster [DBG] pgmap v239: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:28.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:28 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:27.398182+0000 mon.smithi121 (mon.0) 308 : audit [DBG] from='client.? 172.21.15.194:0/3307358817' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:29.975 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.979+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1967551633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 msgr2=0x7f4fb00f4be0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.979+0000 7f4fb7f87700 1 --2- 172.21.15.194:0/1967551633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb00f4be0 secure :-1 s=READY pgs=421 cs=0 l=1 rev1=1 rx=0x7f4fa0009fb0 tx=0x7f4fa0009b30).stop 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.979+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1967551633 shutdown_connections 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.979+0000 7f4fb7f87700 1 --2- 172.21.15.194:0/1967551633 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb00f4be0 unknown :-1 s=CLOSED pgs=421 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.979+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1967551633 >> 172.21.15.194:0/1967551633 conn(0x7f4fb00f02e0 msgr2=0x7f4fb00f2700 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.979+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1967551633 shutdown_connections 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.979+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1967551633 wait complete. 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb7f87700 1 Processor -- start 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb7f87700 1 -- start start 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb7f87700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb0106700 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb7f87700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4fb0106bf0 con 0x7f4fb00f47c0 2024-01-27T23:33:29.976 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb5d23700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb0106700 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb5d23700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb0106700 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55486/0 (socket says 172.21.15.194:55486) 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb5d23700 1 -- 172.21.15.194:0/1117702800 learned_addr learned my addr 172.21.15.194:0/1117702800 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb5d23700 1 -- 172.21.15.194:0/1117702800 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4fa0005040 con 0x7f4fb00f47c0 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.980+0000 7f4fb5d23700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb0106700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4fa0013510 tx=0x7f4fa000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.981+0000 7f4fa67fc700 1 -- 172.21.15.194:0/1117702800 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fa0009320 con 0x7f4fb00f47c0 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.981+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4fb0106df0 con 0x7f4fb00f47c0 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.981+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4fb0107280 con 0x7f4fb00f47c0 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.981+0000 7f4fa67fc700 1 -- 172.21.15.194:0/1117702800 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4fa0013f00 con 0x7f4fb00f47c0 2024-01-27T23:33:29.977 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.981+0000 7f4fa67fc700 1 -- 172.21.15.194:0/1117702800 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4fa0014d60 con 0x7f4fb00f47c0 2024-01-27T23:33:29.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.981+0000 7f4fa67fc700 1 -- 172.21.15.194:0/1117702800 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4fa00263a0 con 0x7f4fb00f47c0 2024-01-27T23:33:29.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.981+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4fb0045ad0 con 0x7f4fb00f47c0 2024-01-27T23:33:29.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.982+0000 7f4fa67fc700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4f9c02ff60 0x7f4f9c032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:29.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.982+0000 7f4fa67fc700 1 -- 172.21.15.194:0/1117702800 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4fa0041380 con 0x7f4fb00f47c0 2024-01-27T23:33:29.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.982+0000 7f4fb5522700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4f9c02ff60 0x7f4f9c032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:29.978 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.982+0000 7f4fb5522700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4f9c02ff60 0x7f4f9c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4fac0096a0 tx=0x7f4fac006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:29.981 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:29.986+0000 7f4fa67fc700 1 -- 172.21.15.194:0/1117702800 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4fa0012920 con 0x7f4fb00f47c0 2024-01-27T23:33:30.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.193+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4fb01080c0 con 0x7f4fb00f47c0 2024-01-27T23:33:30.188 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.193+0000 7f4fa67fc700 1 -- 172.21.15.194:0/1117702800 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4fa0012330 con 0x7f4fb00f47c0 2024-01-27T23:33:30.189 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:30.189 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:30.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4f9c02ff60 msgr2=0x7f4f9c032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:30.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4f9c02ff60 0x7f4f9c032420 secure :-1 s=READY pgs=177 cs=0 l=1 rev1=1 rx=0x7f4fac0096a0 tx=0x7f4fac006b40).stop 2024-01-27T23:33:30.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 msgr2=0x7f4fb0106700 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:30.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb0106700 secure :-1 s=READY pgs=422 cs=0 l=1 rev1=1 rx=0x7f4fa0013510 tx=0x7f4fa000b040).stop 2024-01-27T23:33:30.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 shutdown_connections 2024-01-27T23:33:30.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4f9c02ff60 0x7f4f9c032420 unknown :-1 s=CLOSED pgs=177 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:30.190 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 --2- 172.21.15.194:0/1117702800 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4fb00f47c0 0x7f4fb0106700 unknown :-1 s=CLOSED pgs=422 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:30.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 >> 172.21.15.194:0/1117702800 conn(0x7f4fb00f02e0 msgr2=0x7f4fb00f0db0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:30.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 shutdown_connections 2024-01-27T23:33:30.191 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:30.195+0000 7f4fb7f87700 1 -- 172.21.15.194:0/1117702800 wait complete. 2024-01-27T23:33:30.191 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:30.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:29.334490+0000 mgr.smithi121.hcppvz (mgr.14162) 254 : cluster [DBG] pgmap v240: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:30.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:30 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:30.194952+0000 mon.smithi121 (mon.0) 309 : audit [DBG] from='client.? 172.21.15.194:0/1117702800' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:31.582 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:31.582 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:32.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:32 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:31.334848+0000 mgr.smithi121.hcppvz (mgr.14162) 255 : cluster [DBG] pgmap v241: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.729+0000 7fbc376ed700 1 -- 172.21.15.194:0/3978222764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 msgr2=0x7fbc300f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.729+0000 7fbc376ed700 1 --2- 172.21.15.194:0/3978222764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc300f5830 secure :-1 s=READY pgs=423 cs=0 l=1 rev1=1 rx=0x7fbc20009fb0 tx=0x7fbc20009b30).stop 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.729+0000 7fbc376ed700 1 -- 172.21.15.194:0/3978222764 shutdown_connections 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.729+0000 7fbc376ed700 1 --2- 172.21.15.194:0/3978222764 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc300f5830 unknown :-1 s=CLOSED pgs=423 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.729+0000 7fbc376ed700 1 -- 172.21.15.194:0/3978222764 >> 172.21.15.194:0/3978222764 conn(0x7fbc300f0b70 msgr2=0x7fbc300f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.729+0000 7fbc376ed700 1 -- 172.21.15.194:0/3978222764 shutdown_connections 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.729+0000 7fbc376ed700 1 -- 172.21.15.194:0/3978222764 wait complete. 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.730+0000 7fbc376ed700 1 Processor -- start 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.730+0000 7fbc376ed700 1 -- start start 2024-01-27T23:33:32.725 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.730+0000 7fbc376ed700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc301093f0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:32.726 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.730+0000 7fbc376ed700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fbc30060280 con 0x7fbc300f5410 2024-01-27T23:33:32.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.730+0000 7fbc35489700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc301093f0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:32.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.730+0000 7fbc35489700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc301093f0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55496/0 (socket says 172.21.15.194:55496) 2024-01-27T23:33:32.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.730+0000 7fbc35489700 1 -- 172.21.15.194:0/46720232 learned_addr learned my addr 172.21.15.194:0/46720232 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:32.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.731+0000 7fbc35489700 1 -- 172.21.15.194:0/46720232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fbc20005040 con 0x7fbc300f5410 2024-01-27T23:33:32.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.731+0000 7fbc35489700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc301093f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbc20014000 tx=0x7fbc2000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:32.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.731+0000 7fbc25ffb700 1 -- 172.21.15.194:0/46720232 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc200095f0 con 0x7fbc300f5410 2024-01-27T23:33:32.727 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.731+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fbc30060480 con 0x7fbc300f5410 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.731+0000 7fbc25ffb700 1 -- 172.21.15.194:0/46720232 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fbc20007e70 con 0x7fbc300f5410 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.731+0000 7fbc25ffb700 1 -- 172.21.15.194:0/46720232 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fbc20014ea0 con 0x7fbc300f5410 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.731+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fbc30109b20 con 0x7fbc300f5410 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.732+0000 7fbc25ffb700 1 -- 172.21.15.194:0/46720232 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fbc20025070 con 0x7fbc300f5410 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.732+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fbc30045ad0 con 0x7fbc300f5410 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.732+0000 7fbc25ffb700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbc1c02ffb0 0x7fbc1c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.732+0000 7fbc25ffb700 1 -- 172.21.15.194:0/46720232 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fbc20040a30 con 0x7fbc300f5410 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.732+0000 7fbc34c88700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbc1c02ffb0 0x7fbc1c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:32.728 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.733+0000 7fbc34c88700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbc1c02ffb0 0x7fbc1c032470 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbc2c0096a0 tx=0x7fbc2c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:32.732 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.737+0000 7fbc25ffb700 1 -- 172.21.15.194:0/46720232 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fbc200079c0 con 0x7fbc300f5410 2024-01-27T23:33:32.940 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.945+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fbc3010a340 con 0x7fbc300f5410 2024-01-27T23:33:32.941 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.946+0000 7fbc25ffb700 1 -- 172.21.15.194:0/46720232 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fbc2000f300 con 0x7fbc300f5410 2024-01-27T23:33:32.941 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:32.941 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbc1c02ffb0 msgr2=0x7fbc1c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbc1c02ffb0 0x7fbc1c032470 secure :-1 s=READY pgs=178 cs=0 l=1 rev1=1 rx=0x7fbc2c0096a0 tx=0x7fbc2c006b40).stop 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 msgr2=0x7fbc301093f0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc301093f0 secure :-1 s=READY pgs=424 cs=0 l=1 rev1=1 rx=0x7fbc20014000 tx=0x7fbc2000b040).stop 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 shutdown_connections 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fbc1c02ffb0 0x7fbc1c032470 unknown :-1 s=CLOSED pgs=178 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 --2- 172.21.15.194:0/46720232 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fbc300f5410 0x7fbc301093f0 unknown :-1 s=CLOSED pgs=424 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:32.943 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 >> 172.21.15.194:0/46720232 conn(0x7fbc300f0b70 msgr2=0x7fbc300f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:32.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 shutdown_connections 2024-01-27T23:33:32.944 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:32.948+0000 7fbc376ed700 1 -- 172.21.15.194:0/46720232 wait complete. 2024-01-27T23:33:32.944 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:33.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:33 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:32.947588+0000 mon.smithi121 (mon.0) 310 : audit [DBG] from='client.? 172.21.15.194:0/46720232' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:34.334 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:34.335 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:34.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:34 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:33.335228+0000 mgr.smithi121.hcppvz (mgr.14162) 256 : cluster [DBG] pgmap v242: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.498+0000 7fb8cedf7700 1 -- 172.21.15.194:0/1742101165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 msgr2=0x7fb8c80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.498+0000 7fb8cedf7700 1 --2- 172.21.15.194:0/1742101165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c80f55c0 secure :-1 s=READY pgs=425 cs=0 l=1 rev1=1 rx=0x7fb8c4009fb0 tx=0x7fb8c4009b30).stop 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.498+0000 7fb8cedf7700 1 -- 172.21.15.194:0/1742101165 shutdown_connections 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.498+0000 7fb8cedf7700 1 --2- 172.21.15.194:0/1742101165 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c80f55c0 unknown :-1 s=CLOSED pgs=425 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.498+0000 7fb8cedf7700 1 -- 172.21.15.194:0/1742101165 >> 172.21.15.194:0/1742101165 conn(0x7fb8c80f09a0 msgr2=0x7fb8c80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.498+0000 7fb8cedf7700 1 -- 172.21.15.194:0/1742101165 shutdown_connections 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.498+0000 7fb8cedf7700 1 -- 172.21.15.194:0/1742101165 wait complete. 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.499+0000 7fb8cedf7700 1 Processor -- start 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.499+0000 7fb8cedf7700 1 -- start start 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.499+0000 7fb8cedf7700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:35.494 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.499+0000 7fb8cedf7700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fb8c81058a0 con 0x7fb8c80f51a0 2024-01-27T23:33:35.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.499+0000 7fb8cddf5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:35.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.499+0000 7fb8cddf5700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55180/0 (socket says 172.21.15.194:55180) 2024-01-27T23:33:35.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.499+0000 7fb8cddf5700 1 -- 172.21.15.194:0/2787736854 learned_addr learned my addr 172.21.15.194:0/2787736854 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:35.496 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.500+0000 7fb8cddf5700 1 -- 172.21.15.194:0/2787736854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fb8c4005040 con 0x7fb8c80f51a0 2024-01-27T23:33:35.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.500+0000 7fb8cddf5700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81053b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fb8c4013510 tx=0x7fb8c400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:35.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.500+0000 7fb8be7fc700 1 -- 172.21.15.194:0/2787736854 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c4009320 con 0x7fb8c80f51a0 2024-01-27T23:33:35.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.500+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fb8c8105aa0 con 0x7fb8c80f51a0 2024-01-27T23:33:35.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.500+0000 7fb8be7fc700 1 -- 172.21.15.194:0/2787736854 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fb8c4013f00 con 0x7fb8c80f51a0 2024-01-27T23:33:35.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.500+0000 7fb8be7fc700 1 -- 172.21.15.194:0/2787736854 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fb8c4014dc0 con 0x7fb8c80f51a0 2024-01-27T23:33:35.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.500+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fb8c8105f30 con 0x7fb8c80f51a0 2024-01-27T23:33:35.497 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.501+0000 7fb8be7fc700 1 -- 172.21.15.194:0/2787736854 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fb8c40095f0 con 0x7fb8c80f51a0 2024-01-27T23:33:35.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.501+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fb8c8045ad0 con 0x7fb8c80f51a0 2024-01-27T23:33:35.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.501+0000 7fb8be7fc700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb8b402ff60 0x7fb8b4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:35.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.501+0000 7fb8be7fc700 1 -- 172.21.15.194:0/2787736854 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fb8c40410c0 con 0x7fb8c80f51a0 2024-01-27T23:33:35.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.501+0000 7fb8cd5f4700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb8b402ff60 0x7fb8b4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:35.498 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.501+0000 7fb8cd5f4700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb8b402ff60 0x7fb8b4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb8b80096a0 tx=0x7fb8b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:35.500 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.505+0000 7fb8be7fc700 1 -- 172.21.15.194:0/2787736854 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fb8c4014420 con 0x7fb8c80f51a0 2024-01-27T23:33:35.708 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.713+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fb8c8106de0 con 0x7fb8c80f51a0 2024-01-27T23:33:35.709 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.714+0000 7fb8be7fc700 1 -- 172.21.15.194:0/2787736854 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fb8c4010230 con 0x7fb8c80f51a0 2024-01-27T23:33:35.709 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:35.709 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb8b402ff60 msgr2=0x7fb8b4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb8b402ff60 0x7fb8b4032420 secure :-1 s=READY pgs=179 cs=0 l=1 rev1=1 rx=0x7fb8b80096a0 tx=0x7fb8b8006b40).stop 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 msgr2=0x7fb8c81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81053b0 secure :-1 s=READY pgs=426 cs=0 l=1 rev1=1 rx=0x7fb8c4013510 tx=0x7fb8c400b040).stop 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 shutdown_connections 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fb8b402ff60 0x7fb8b4032420 unknown :-1 s=CLOSED pgs=179 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 --2- 172.21.15.194:0/2787736854 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fb8c80f51a0 0x7fb8c81053b0 unknown :-1 s=CLOSED pgs=426 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 >> 172.21.15.194:0/2787736854 conn(0x7fb8c80f09a0 msgr2=0x7fb8c80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 shutdown_connections 2024-01-27T23:33:35.711 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:35.716+0000 7fb8cedf7700 1 -- 172.21.15.194:0/2787736854 wait complete. 2024-01-27T23:33:35.712 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:36.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:35.335554+0000 mgr.smithi121.hcppvz (mgr.14162) 257 : cluster [DBG] pgmap v243: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:36.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:36 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:35.715542+0000 mon.smithi121 (mon.0) 311 : audit [DBG] from='client.? 172.21.15.194:0/2787736854' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:37.137 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:37.137 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:38.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.373+0000 7fc3ccd12700 1 -- 172.21.15.194:0/1463471260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 msgr2=0x7fc3c80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:38.369 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.373+0000 7fc3ccd12700 1 --2- 172.21.15.194:0/1463471260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c80f5830 secure :-1 s=READY pgs=427 cs=0 l=1 rev1=1 rx=0x7fc3bc009fb0 tx=0x7fc3bc009b30).stop 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.373+0000 7fc3ccd12700 1 -- 172.21.15.194:0/1463471260 shutdown_connections 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.373+0000 7fc3ccd12700 1 --2- 172.21.15.194:0/1463471260 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c80f5830 unknown :-1 s=CLOSED pgs=427 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.373+0000 7fc3ccd12700 1 -- 172.21.15.194:0/1463471260 >> 172.21.15.194:0/1463471260 conn(0x7fc3c80f0b70 msgr2=0x7fc3c80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.373+0000 7fc3ccd12700 1 -- 172.21.15.194:0/1463471260 shutdown_connections 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.373+0000 7fc3ccd12700 1 -- 172.21.15.194:0/1463471260 wait complete. 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3ccd12700 1 Processor -- start 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3ccd12700 1 -- start start 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3ccd12700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c8106720 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3ccd12700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fc3c8106c10 con 0x7fc3c80f5410 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3c659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c8106720 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3c659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c8106720 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55198/0 (socket says 172.21.15.194:55198) 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3c659c700 1 -- 172.21.15.194:0/2962277649 learned_addr learned my addr 172.21.15.194:0/2962277649 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3c659c700 1 -- 172.21.15.194:0/2962277649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fc3bc005040 con 0x7fc3c80f5410 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.374+0000 7fc3c659c700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c8106720 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fc3bc013510 tx=0x7fc3bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:38.370 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.375+0000 7fc3b6ffd700 1 -- 172.21.15.194:0/2962277649 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3bc009320 con 0x7fc3c80f5410 2024-01-27T23:33:38.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.375+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fc3c8106e10 con 0x7fc3c80f5410 2024-01-27T23:33:38.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.375+0000 7fc3b6ffd700 1 -- 172.21.15.194:0/2962277649 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fc3bc013f00 con 0x7fc3c80f5410 2024-01-27T23:33:38.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.375+0000 7fc3b6ffd700 1 -- 172.21.15.194:0/2962277649 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fc3bc014d90 con 0x7fc3c80f5410 2024-01-27T23:33:38.371 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.375+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fc3c81072a0 con 0x7fc3c80f5410 2024-01-27T23:33:38.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.375+0000 7fc3b6ffd700 1 -- 172.21.15.194:0/2962277649 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fc3bc0095f0 con 0x7fc3c80f5410 2024-01-27T23:33:38.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.375+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fc3c8045ad0 con 0x7fc3c80f5410 2024-01-27T23:33:38.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.376+0000 7fc3b6ffd700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc3b002ff60 0x7fc3b0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:38.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.376+0000 7fc3b6ffd700 1 -- 172.21.15.194:0/2962277649 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fc3bc0410c0 con 0x7fc3c80f5410 2024-01-27T23:33:38.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.376+0000 7fc3c5d9b700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc3b002ff60 0x7fc3b0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:38.372 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.376+0000 7fc3c5d9b700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc3b002ff60 0x7fc3b0032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fc3b80096a0 tx=0x7fc3b8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:38.375 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.381+0000 7fc3b6ffd700 1 -- 172.21.15.194:0/2962277649 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fc3bc01d7f0 con 0x7fc3c80f5410 2024-01-27T23:33:38.580 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.585+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fc3c8108150 con 0x7fc3c80f5410 2024-01-27T23:33:38.581 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.586+0000 7fc3b6ffd700 1 -- 172.21.15.194:0/2962277649 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fc3bc010230 con 0x7fc3c80f5410 2024-01-27T23:33:38.581 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:38.581 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc3b002ff60 msgr2=0x7fc3b0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc3b002ff60 0x7fc3b0032420 secure :-1 s=READY pgs=180 cs=0 l=1 rev1=1 rx=0x7fc3b80096a0 tx=0x7fc3b8006b40).stop 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 msgr2=0x7fc3c8106720 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c8106720 secure :-1 s=READY pgs=428 cs=0 l=1 rev1=1 rx=0x7fc3bc013510 tx=0x7fc3bc00b040).stop 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 shutdown_connections 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fc3b002ff60 0x7fc3b0032420 unknown :-1 s=CLOSED pgs=180 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 --2- 172.21.15.194:0/2962277649 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fc3c80f5410 0x7fc3c8106720 unknown :-1 s=CLOSED pgs=428 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 >> 172.21.15.194:0/2962277649 conn(0x7fc3c80f0b70 msgr2=0x7fc3c805fe20 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 shutdown_connections 2024-01-27T23:33:38.583 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:38.588+0000 7fc3ccd12700 1 -- 172.21.15.194:0/2962277649 wait complete. 2024-01-27T23:33:38.584 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:38.843 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:38 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:37.335896+0000 mgr.smithi121.hcppvz (mgr.14162) 258 : cluster [DBG] pgmap v244: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:39.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:39 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:38.587451+0000 mon.smithi121 (mon.0) 312 : audit [DBG] from='client.? 172.21.15.194:0/2962277649' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:40.000 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:40.000 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:40.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:40 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:39.336134+0000 mgr.smithi121.hcppvz (mgr.14162) 259 : cluster [DBG] pgmap v245: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:41.114 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.119+0000 7f4eefb5d700 1 -- 172.21.15.194:0/1907158955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 msgr2=0x7f4ee80f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:41.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.119+0000 7f4eefb5d700 1 --2- 172.21.15.194:0/1907158955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee80f5830 secure :-1 s=READY pgs=429 cs=0 l=1 rev1=1 rx=0x7f4ee4009fb0 tx=0x7f4ee4009b30).stop 2024-01-27T23:33:41.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.119+0000 7f4eefb5d700 1 -- 172.21.15.194:0/1907158955 shutdown_connections 2024-01-27T23:33:41.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.119+0000 7f4eefb5d700 1 --2- 172.21.15.194:0/1907158955 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee80f5830 unknown :-1 s=CLOSED pgs=429 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:41.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.119+0000 7f4eefb5d700 1 -- 172.21.15.194:0/1907158955 >> 172.21.15.194:0/1907158955 conn(0x7f4ee80f0b70 msgr2=0x7f4ee80f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:41.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.120+0000 7f4eefb5d700 1 -- 172.21.15.194:0/1907158955 shutdown_connections 2024-01-27T23:33:41.115 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.120+0000 7f4eefb5d700 1 -- 172.21.15.194:0/1907158955 wait complete. 2024-01-27T23:33:41.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.120+0000 7f4eefb5d700 1 Processor -- start 2024-01-27T23:33:41.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.120+0000 7f4eefb5d700 1 -- start start 2024-01-27T23:33:41.116 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.120+0000 7f4eefb5d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee8109390 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.120+0000 7f4eefb5d700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f4ee8060280 con 0x7f4ee80f5410 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4eed8f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee8109390 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4eed8f9700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee8109390 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55216/0 (socket says 172.21.15.194:55216) 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4eed8f9700 1 -- 172.21.15.194:0/2964989060 learned_addr learned my addr 172.21.15.194:0/2964989060 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4eed8f9700 1 -- 172.21.15.194:0/2964989060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f4ee4005040 con 0x7f4ee80f5410 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4eed8f9700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee8109390 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f4ee4013510 tx=0x7f4ee400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4ede7fc700 1 -- 172.21.15.194:0/2964989060 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee4009320 con 0x7f4ee80f5410 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f4ee8060480 con 0x7f4ee80f5410 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4ede7fc700 1 -- 172.21.15.194:0/2964989060 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f4ee4013f00 con 0x7f4ee80f5410 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4ede7fc700 1 -- 172.21.15.194:0/2964989060 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f4ee4014d90 con 0x7f4ee80f5410 2024-01-27T23:33:41.117 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.121+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f4ee8109ac0 con 0x7f4ee80f5410 2024-01-27T23:33:41.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.122+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f4ee8045ad0 con 0x7f4ee80f5410 2024-01-27T23:33:41.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.122+0000 7f4ede7fc700 1 -- 172.21.15.194:0/2964989060 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f4ee40095f0 con 0x7f4ee80f5410 2024-01-27T23:33:41.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.122+0000 7f4ede7fc700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ed402ff60 0x7f4ed4032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:41.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.122+0000 7f4ede7fc700 1 -- 172.21.15.194:0/2964989060 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f4ee4041000 con 0x7f4ee80f5410 2024-01-27T23:33:41.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.122+0000 7f4eed0f8700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ed402ff60 0x7f4ed4032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:41.118 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.123+0000 7f4eed0f8700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ed402ff60 0x7f4ed4032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4ed80096a0 tx=0x7f4ed8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:41.121 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.127+0000 7f4ede7fc700 1 -- 172.21.15.194:0/2964989060 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f4ee4014420 con 0x7f4ee80f5410 2024-01-27T23:33:41.330 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.335+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f4ee80fae60 con 0x7f4ee80f5410 2024-01-27T23:33:41.331 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.336+0000 7f4ede7fc700 1 -- 172.21.15.194:0/2964989060 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f4ee4010230 con 0x7f4ee80f5410 2024-01-27T23:33:41.331 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:41.331 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:41.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ed402ff60 msgr2=0x7f4ed4032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:41.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ed402ff60 0x7f4ed4032420 secure :-1 s=READY pgs=181 cs=0 l=1 rev1=1 rx=0x7f4ed80096a0 tx=0x7f4ed8006b40).stop 2024-01-27T23:33:41.332 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 msgr2=0x7f4ee8109390 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee8109390 secure :-1 s=READY pgs=430 cs=0 l=1 rev1=1 rx=0x7f4ee4013510 tx=0x7f4ee400b040).stop 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 shutdown_connections 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f4ed402ff60 0x7f4ed4032420 unknown :-1 s=CLOSED pgs=181 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 --2- 172.21.15.194:0/2964989060 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f4ee80f5410 0x7f4ee8109390 unknown :-1 s=CLOSED pgs=430 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 >> 172.21.15.194:0/2964989060 conn(0x7f4ee80f0b70 msgr2=0x7f4ee80f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 shutdown_connections 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:41.337+0000 7f4eefb5d700 1 -- 172.21.15.194:0/2964989060 wait complete. 2024-01-27T23:33:41.333 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:42.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:41.336372+0000 mgr.smithi121.hcppvz (mgr.14162) 260 : cluster [DBG] pgmap v246: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:42.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:42 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:41.336993+0000 mon.smithi121 (mon.0) 313 : audit [DBG] from='client.? 172.21.15.194:0/2964989060' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:42.722 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:42.722 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:43.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 -- 172.21.15.194:0/438744363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 msgr2=0x7f9da80f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:43.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 --2- 172.21.15.194:0/438744363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da80f55c0 secure :-1 s=READY pgs=431 cs=0 l=1 rev1=1 rx=0x7f9d9c009fb0 tx=0x7f9d9c009b30).stop 2024-01-27T23:33:43.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 -- 172.21.15.194:0/438744363 shutdown_connections 2024-01-27T23:33:43.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 --2- 172.21.15.194:0/438744363 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da80f55c0 unknown :-1 s=CLOSED pgs=431 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:43.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 -- 172.21.15.194:0/438744363 >> 172.21.15.194:0/438744363 conn(0x7f9da80f09a0 msgr2=0x7f9da80f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:43.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 -- 172.21.15.194:0/438744363 shutdown_connections 2024-01-27T23:33:43.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 -- 172.21.15.194:0/438744363 wait complete. 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 Processor -- start 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.923+0000 7f9da759e700 1 -- start start 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.924+0000 7f9da759e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da81053b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.924+0000 7f9da759e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9da81058a0 con 0x7f9da80f51a0 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.924+0000 7f9da659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da81053b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.924+0000 7f9da659c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da81053b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:55238/0 (socket says 172.21.15.194:55238) 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.924+0000 7f9da659c700 1 -- 172.21.15.194:0/2025078002 learned_addr learned my addr 172.21.15.194:0/2025078002 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.924+0000 7f9da659c700 1 -- 172.21.15.194:0/2025078002 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9d9c005040 con 0x7f9da80f51a0 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.924+0000 7f9da659c700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da81053b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f9d9c013510 tx=0x7f9d9c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.925+0000 7f9d96ffd700 1 -- 172.21.15.194:0/2025078002 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9c009320 con 0x7f9da80f51a0 2024-01-27T23:33:43.920 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.925+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9da8105aa0 con 0x7f9da80f51a0 2024-01-27T23:33:43.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.925+0000 7f9d96ffd700 1 -- 172.21.15.194:0/2025078002 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9d9c013f00 con 0x7f9da80f51a0 2024-01-27T23:33:43.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.925+0000 7f9d96ffd700 1 -- 172.21.15.194:0/2025078002 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9d9c014dc0 con 0x7f9da80f51a0 2024-01-27T23:33:43.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.925+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9da8105f30 con 0x7f9da80f51a0 2024-01-27T23:33:43.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.925+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9da8045ad0 con 0x7f9da80f51a0 2024-01-27T23:33:43.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.926+0000 7f9d96ffd700 1 -- 172.21.15.194:0/2025078002 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9d9c0095f0 con 0x7f9da80f51a0 2024-01-27T23:33:43.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.926+0000 7f9d96ffd700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9d9002ff60 0x7f9d90032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:43.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.926+0000 7f9d96ffd700 1 -- 172.21.15.194:0/2025078002 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9d9c042e50 con 0x7f9da80f51a0 2024-01-27T23:33:43.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.926+0000 7f9da5d9b700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9d9002ff60 0x7f9d90032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:43.922 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.926+0000 7f9da5d9b700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9d9002ff60 0x7f9d90032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9d980096a0 tx=0x7f9d98006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:43.925 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:43.930+0000 7f9d96ffd700 1 -- 172.21.15.194:0/2025078002 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9d9c014420 con 0x7f9da80f51a0 2024-01-27T23:33:44.132 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.138+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9da8106d80 con 0x7f9da80f51a0 2024-01-27T23:33:44.133 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.138+0000 7f9d96ffd700 1 -- 172.21.15.194:0/2025078002 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9d9c01f980 con 0x7f9da80f51a0 2024-01-27T23:33:44.133 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:44.133 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9d9002ff60 msgr2=0x7f9d90032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9d9002ff60 0x7f9d90032420 secure :-1 s=READY pgs=182 cs=0 l=1 rev1=1 rx=0x7f9d980096a0 tx=0x7f9d98006b40).stop 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 msgr2=0x7f9da81053b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da81053b0 secure :-1 s=READY pgs=432 cs=0 l=1 rev1=1 rx=0x7f9d9c013510 tx=0x7f9d9c00b040).stop 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 shutdown_connections 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9d9002ff60 0x7f9d90032420 unknown :-1 s=CLOSED pgs=182 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 --2- 172.21.15.194:0/2025078002 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9da80f51a0 0x7f9da81053b0 unknown :-1 s=CLOSED pgs=432 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 >> 172.21.15.194:0/2025078002 conn(0x7f9da80f09a0 msgr2=0x7f9da80f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 shutdown_connections 2024-01-27T23:33:44.135 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:44.140+0000 7f9da759e700 1 -- 172.21.15.194:0/2025078002 wait complete. 2024-01-27T23:33:44.136 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:44.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:43.336698+0000 mgr.smithi121.hcppvz (mgr.14162) 261 : cluster [DBG] pgmap v247: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:44.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:44 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:44.139636+0000 mon.smithi121 (mon.0) 314 : audit [DBG] from='client.? 172.21.15.194:0/2025078002' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:45.519 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:45.519 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 -- 172.21.15.194:0/1571621337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 msgr2=0x7f75040f55c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 --2- 172.21.15.194:0/1571621337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f75040f55c0 secure :-1 s=READY pgs=433 cs=0 l=1 rev1=1 rx=0x7f74f4009fb0 tx=0x7f74f4009b30).stop 2024-01-27T23:33:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 -- 172.21.15.194:0/1571621337 shutdown_connections 2024-01-27T23:33:46.656 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 --2- 172.21.15.194:0/1571621337 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f75040f55c0 unknown :-1 s=CLOSED pgs=433 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:46.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 -- 172.21.15.194:0/1571621337 >> 172.21.15.194:0/1571621337 conn(0x7f75040f09a0 msgr2=0x7f75040f2dc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:46.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 -- 172.21.15.194:0/1571621337 shutdown_connections 2024-01-27T23:33:46.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 -- 172.21.15.194:0/1571621337 wait complete. 2024-01-27T23:33:46.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 Processor -- start 2024-01-27T23:33:46.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.661+0000 7f7508e9e700 1 -- start start 2024-01-27T23:33:46.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.662+0000 7f7508e9e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f7504109170 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:46.657 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.662+0000 7f7508e9e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7504060280 con 0x7f75040f51a0 2024-01-27T23:33:46.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f75037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f7504109170 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:46.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f75037fe700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f7504109170 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:54938/0 (socket says 172.21.15.194:54938) 2024-01-27T23:33:46.658 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f75037fe700 1 -- 172.21.15.194:0/384060194 learned_addr learned my addr 172.21.15.194:0/384060194 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:46.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f75037fe700 1 -- 172.21.15.194:0/384060194 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f74f4005040 con 0x7f75040f51a0 2024-01-27T23:33:46.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f75037fe700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f7504109170 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f74f4013510 tx=0x7f74f400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:46.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f74ebfff700 1 -- 172.21.15.194:0/384060194 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74f4009320 con 0x7f75040f51a0 2024-01-27T23:33:46.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7504060480 con 0x7f75040f51a0 2024-01-27T23:33:46.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f74ebfff700 1 -- 172.21.15.194:0/384060194 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f74f4013f00 con 0x7f75040f51a0 2024-01-27T23:33:46.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f74ebfff700 1 -- 172.21.15.194:0/384060194 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f74f4014dc0 con 0x7f75040f51a0 2024-01-27T23:33:46.659 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.663+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f75041098a0 con 0x7f75040f51a0 2024-01-27T23:33:46.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.664+0000 7f74ebfff700 1 -- 172.21.15.194:0/384060194 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f74f40095f0 con 0x7f75040f51a0 2024-01-27T23:33:46.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.664+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7504045ad0 con 0x7f75040f51a0 2024-01-27T23:33:46.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.664+0000 7f74ebfff700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f74ec02ff60 0x7f74ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:46.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.664+0000 7f74ebfff700 1 -- 172.21.15.194:0/384060194 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f74f4041020 con 0x7f75040f51a0 2024-01-27T23:33:46.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.665+0000 7f7502ffd700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f74ec02ff60 0x7f74ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:46.660 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.665+0000 7f7502ffd700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f74ec02ff60 0x7f74ec032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f74f80096a0 tx=0x7f74f8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:46.664 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.669+0000 7f74ebfff700 1 -- 172.21.15.194:0/384060194 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f74f4014420 con 0x7f75040f51a0 2024-01-27T23:33:46.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:46 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:45.337005+0000 mgr.smithi121.hcppvz (mgr.14162) 262 : cluster [DBG] pgmap v248: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:46.871 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.876+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f75040fabe0 con 0x7f75040f51a0 2024-01-27T23:33:46.872 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.877+0000 7f74ebfff700 1 -- 172.21.15.194:0/384060194 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f74f401c920 con 0x7f75040f51a0 2024-01-27T23:33:46.872 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:46.872 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f74ec02ff60 msgr2=0x7f74ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f74ec02ff60 0x7f74ec032420 secure :-1 s=READY pgs=183 cs=0 l=1 rev1=1 rx=0x7f74f80096a0 tx=0x7f74f8006b40).stop 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 msgr2=0x7f7504109170 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f7504109170 secure :-1 s=READY pgs=434 cs=0 l=1 rev1=1 rx=0x7f74f4013510 tx=0x7f74f400b040).stop 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 shutdown_connections 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f74ec02ff60 0x7f74ec032420 unknown :-1 s=CLOSED pgs=183 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 --2- 172.21.15.194:0/384060194 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f75040f51a0 0x7f7504109170 unknown :-1 s=CLOSED pgs=434 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 >> 172.21.15.194:0/384060194 conn(0x7f75040f09a0 msgr2=0x7f75040f1460 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 shutdown_connections 2024-01-27T23:33:46.874 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:46.879+0000 7f7508e9e700 1 -- 172.21.15.194:0/384060194 wait complete. 2024-01-27T23:33:46.875 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:47.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:47 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:46.878403+0000 mon.smithi121 (mon.0) 315 : audit [DBG] from='client.? 172.21.15.194:0/384060194' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:48.258 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:48.259 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:48.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:48 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:47.337345+0000 mgr.smithi121.hcppvz (mgr.14162) 263 : cluster [DBG] pgmap v249: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:49.392 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.396+0000 7f62d35a4700 1 -- 172.21.15.194:0/2529385592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 msgr2=0x7f62cc0f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.396+0000 7f62d35a4700 1 --2- 172.21.15.194:0/2529385592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc0f5830 secure :-1 s=READY pgs=435 cs=0 l=1 rev1=1 rx=0x7f62bc009fb0 tx=0x7f62bc009b30).stop 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.396+0000 7f62d35a4700 1 -- 172.21.15.194:0/2529385592 shutdown_connections 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.396+0000 7f62d35a4700 1 --2- 172.21.15.194:0/2529385592 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc0f5830 unknown :-1 s=CLOSED pgs=435 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.396+0000 7f62d35a4700 1 -- 172.21.15.194:0/2529385592 >> 172.21.15.194:0/2529385592 conn(0x7f62cc0f0b70 msgr2=0x7f62cc0f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.396+0000 7f62d35a4700 1 -- 172.21.15.194:0/2529385592 shutdown_connections 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.396+0000 7f62d35a4700 1 -- 172.21.15.194:0/2529385592 wait complete. 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.397+0000 7f62d35a4700 1 Processor -- start 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.397+0000 7f62d35a4700 1 -- start start 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.397+0000 7f62d35a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc109430 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.397+0000 7f62d35a4700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f62cc060240 con 0x7f62cc0f5410 2024-01-27T23:33:49.393 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.397+0000 7f62d1340700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc109430 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:49.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.397+0000 7f62d1340700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc109430 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:54960/0 (socket says 172.21.15.194:54960) 2024-01-27T23:33:49.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.397+0000 7f62d1340700 1 -- 172.21.15.194:0/1331784221 learned_addr learned my addr 172.21.15.194:0/1331784221 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:49.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.398+0000 7f62d1340700 1 -- 172.21.15.194:0/1331784221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f62bc005040 con 0x7f62cc0f5410 2024-01-27T23:33:49.394 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.398+0000 7f62d1340700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc109430 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f62bc013510 tx=0x7f62bc00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:49.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.398+0000 7f62c1ffb700 1 -- 172.21.15.194:0/1331784221 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62bc009320 con 0x7f62cc0f5410 2024-01-27T23:33:49.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.398+0000 7f62c1ffb700 1 -- 172.21.15.194:0/1331784221 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f62bc013f00 con 0x7f62cc0f5410 2024-01-27T23:33:49.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.398+0000 7f62c1ffb700 1 -- 172.21.15.194:0/1331784221 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f62bc014d90 con 0x7f62cc0f5410 2024-01-27T23:33:49.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.398+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f62cc060440 con 0x7f62cc0f5410 2024-01-27T23:33:49.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.398+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f62cc109ae0 con 0x7f62cc0f5410 2024-01-27T23:33:49.395 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.399+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f62cc045ad0 con 0x7f62cc0f5410 2024-01-27T23:33:49.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.400+0000 7f62c1ffb700 1 -- 172.21.15.194:0/1331784221 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f62bc0095f0 con 0x7f62cc0f5410 2024-01-27T23:33:49.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.401+0000 7f62c1ffb700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f62b8030000 0x7f62b80324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:49.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.401+0000 7f62c1ffb700 1 -- 172.21.15.194:0/1331784221 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f62bc0411a0 con 0x7f62cc0f5410 2024-01-27T23:33:49.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.401+0000 7f62d0b3f700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f62b8030000 0x7f62b80324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:49.396 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.401+0000 7f62d0b3f700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f62b8030000 0x7f62b80324c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f62c80096a0 tx=0x7f62c8006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:49.399 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.404+0000 7f62c1ffb700 1 -- 172.21.15.194:0/1331784221 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f62bc014420 con 0x7f62cc0f5410 2024-01-27T23:33:49.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.614+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f62cc0faf50 con 0x7f62cc0f5410 2024-01-27T23:33:49.609 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.615+0000 7f62c1ffb700 1 -- 172.21.15.194:0/1331784221 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f62bc00f7b0 con 0x7f62cc0f5410 2024-01-27T23:33:49.610 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:49.610 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f62b8030000 msgr2=0x7f62b80324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f62b8030000 0x7f62b80324c0 secure :-1 s=READY pgs=184 cs=0 l=1 rev1=1 rx=0x7f62c80096a0 tx=0x7f62c8006b40).stop 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 msgr2=0x7f62cc109430 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc109430 secure :-1 s=READY pgs=436 cs=0 l=1 rev1=1 rx=0x7f62bc013510 tx=0x7f62bc00b040).stop 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 shutdown_connections 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f62b8030000 0x7f62b80324c0 unknown :-1 s=CLOSED pgs=184 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 --2- 172.21.15.194:0/1331784221 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f62cc0f5410 0x7f62cc109430 unknown :-1 s=CLOSED pgs=436 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 >> 172.21.15.194:0/1331784221 conn(0x7f62cc0f0b70 msgr2=0x7f62cc0f1660 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.616+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 shutdown_connections 2024-01-27T23:33:49.612 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:49.617+0000 7f62d35a4700 1 -- 172.21.15.194:0/1331784221 wait complete. 2024-01-27T23:33:49.613 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:50.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:49.337656+0000 mgr.smithi121.hcppvz (mgr.14162) 264 : cluster [DBG] pgmap v250: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:50.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:50 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:49.615987+0000 mon.smithi121 (mon.0) 316 : audit [DBG] from='client.? 172.21.15.194:0/1331784221' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:50.991 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:50.991 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:52.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.126+0000 7f99baf38700 1 -- 172.21.15.194:0/2622922211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 msgr2=0x7f99b40f66c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:52.125 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.126+0000 7f99baf38700 1 --2- 172.21.15.194:0/2622922211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b40f66c0 secure :-1 s=READY pgs=437 cs=0 l=1 rev1=1 rx=0x7f99b0009fb0 tx=0x7f99b0009b30).stop 2024-01-27T23:33:52.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.126+0000 7f99baf38700 1 -- 172.21.15.194:0/2622922211 shutdown_connections 2024-01-27T23:33:52.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.126+0000 7f99baf38700 1 --2- 172.21.15.194:0/2622922211 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b40f66c0 unknown :-1 s=CLOSED pgs=437 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:52.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.126+0000 7f99baf38700 1 -- 172.21.15.194:0/2622922211 >> 172.21.15.194:0/2622922211 conn(0x7f99b40f1aa0 msgr2=0x7f99b40f3ec0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:52.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.126+0000 7f99baf38700 1 -- 172.21.15.194:0/2622922211 shutdown_connections 2024-01-27T23:33:52.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.126+0000 7f99baf38700 1 -- 172.21.15.194:0/2622922211 wait complete. 2024-01-27T23:33:52.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99baf38700 1 Processor -- start 2024-01-27T23:33:52.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99baf38700 1 -- start start 2024-01-27T23:33:52.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99baf38700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b41064b0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:52.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99baf38700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f99b41069a0 con 0x7f99b40f62a0 2024-01-27T23:33:52.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99b9f36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b41064b0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:52.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99b9f36700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b41064b0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:54978/0 (socket says 172.21.15.194:54978) 2024-01-27T23:33:52.127 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99b9f36700 1 -- 172.21.15.194:0/665677517 learned_addr learned my addr 172.21.15.194:0/665677517 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99b9f36700 1 -- 172.21.15.194:0/665677517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f99b0005040 con 0x7f99b40f62a0 2024-01-27T23:33:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.127+0000 7f99b9f36700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b41064b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f99b0009830 tx=0x7f99b000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.128+0000 7f99aa7fc700 1 -- 172.21.15.194:0/665677517 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99b0009320 con 0x7f99b40f62a0 2024-01-27T23:33:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.128+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f99b4106ba0 con 0x7f99b40f62a0 2024-01-27T23:33:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.128+0000 7f99aa7fc700 1 -- 172.21.15.194:0/665677517 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f99b00136c0 con 0x7f99b40f62a0 2024-01-27T23:33:52.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.128+0000 7f99aa7fc700 1 -- 172.21.15.194:0/665677517 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f99b001ee90 con 0x7f99b40f62a0 2024-01-27T23:33:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.128+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f99b4107030 con 0x7f99b40f62a0 2024-01-27T23:33:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.128+0000 7f99aa7fc700 1 -- 172.21.15.194:0/665677517 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f99b0014420 con 0x7f99b40f62a0 2024-01-27T23:33:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.129+0000 7f99aa7fc700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f99a002ff10 0x7f99a00323d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.129+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f99b4045ad0 con 0x7f99b40f62a0 2024-01-27T23:33:52.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.129+0000 7f99aa7fc700 1 -- 172.21.15.194:0/665677517 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f99b0040fe0 con 0x7f99b40f62a0 2024-01-27T23:33:52.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.129+0000 7f99b9735700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f99a002ff10 0x7f99a00323d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:52.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.129+0000 7f99b9735700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f99a002ff10 0x7f99a00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f99a40096a0 tx=0x7f99a4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:52.130 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.133+0000 7f99aa7fc700 1 -- 172.21.15.194:0/665677517 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f99b00273f0 con 0x7f99b40f62a0 2024-01-27T23:33:52.335 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.340+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f99b4107ee0 con 0x7f99b40f62a0 2024-01-27T23:33:52.336 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.341+0000 7f99aa7fc700 1 -- 172.21.15.194:0/665677517 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f99b001c320 con 0x7f99b40f62a0 2024-01-27T23:33:52.336 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:52.336 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:52.337 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f99a002ff10 msgr2=0x7f99a00323d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f99a002ff10 0x7f99a00323d0 secure :-1 s=READY pgs=185 cs=0 l=1 rev1=1 rx=0x7f99a40096a0 tx=0x7f99a4006b40).stop 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 msgr2=0x7f99b41064b0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b41064b0 secure :-1 s=READY pgs=438 cs=0 l=1 rev1=1 rx=0x7f99b0009830 tx=0x7f99b000b040).stop 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 shutdown_connections 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f99a002ff10 0x7f99a00323d0 unknown :-1 s=CLOSED pgs=185 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 --2- 172.21.15.194:0/665677517 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f99b40f62a0 0x7f99b41064b0 unknown :-1 s=CLOSED pgs=438 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 >> 172.21.15.194:0/665677517 conn(0x7f99b40f1aa0 msgr2=0x7f99b40f2510 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 shutdown_connections 2024-01-27T23:33:52.338 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:52.342+0000 7f99baf38700 1 -- 172.21.15.194:0/665677517 wait complete. 2024-01-27T23:33:52.339 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:52.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:52 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:51.338002+0000 mgr.smithi121.hcppvz (mgr.14162) 265 : cluster [DBG] pgmap v251: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:53.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:53 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:52.342055+0000 mon.smithi121 (mon.0) 317 : audit [DBG] from='client.? 172.21.15.194:0/665677517' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:53.761 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:53.761 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:54.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:54 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:53.338413+0000 mgr.smithi121.hcppvz (mgr.14162) 266 : cluster [DBG] pgmap v252: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:54.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.951+0000 7f9de7308700 1 -- 172.21.15.194:0/1490183811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 msgr2=0x7f9de00f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:54.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.951+0000 7f9de7308700 1 --2- 172.21.15.194:0/1490183811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de00f6930 secure :-1 s=READY pgs=439 cs=0 l=1 rev1=1 rx=0x7f9dd0009fb0 tx=0x7f9dd0009b30).stop 2024-01-27T23:33:54.950 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.951+0000 7f9de7308700 1 -- 172.21.15.194:0/1490183811 shutdown_connections 2024-01-27T23:33:54.951 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.951+0000 7f9de7308700 1 --2- 172.21.15.194:0/1490183811 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de00f6930 unknown :-1 s=CLOSED pgs=439 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:54.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.951+0000 7f9de7308700 1 -- 172.21.15.194:0/1490183811 >> 172.21.15.194:0/1490183811 conn(0x7f9de00f1c70 msgr2=0x7f9de00f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:54.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.951+0000 7f9de7308700 1 -- 172.21.15.194:0/1490183811 shutdown_connections 2024-01-27T23:33:54.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.951+0000 7f9de7308700 1 -- 172.21.15.194:0/1490183811 wait complete. 2024-01-27T23:33:54.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de7308700 1 Processor -- start 2024-01-27T23:33:54.952 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de7308700 1 -- start start 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de7308700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de0106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de7308700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f9de0106c20 con 0x7f9de00f6510 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de50a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de0106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de50a4700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de0106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:54998/0 (socket says 172.21.15.194:54998) 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de50a4700 1 -- 172.21.15.194:0/2949915694 learned_addr learned my addr 172.21.15.194:0/2949915694 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.952+0000 7f9de50a4700 1 -- 172.21.15.194:0/2949915694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f9dd0005040 con 0x7f9de00f6510 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.953+0000 7f9de50a4700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de0106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9dd0013510 tx=0x7f9dd000b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.953+0000 7f9dd5ffb700 1 -- 172.21.15.194:0/2949915694 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dd0009320 con 0x7f9de00f6510 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.953+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f9de0106e20 con 0x7f9de00f6510 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.953+0000 7f9dd5ffb700 1 -- 172.21.15.194:0/2949915694 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f9dd0013f00 con 0x7f9de00f6510 2024-01-27T23:33:54.953 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.953+0000 7f9dd5ffb700 1 -- 172.21.15.194:0/2949915694 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f9dd0014d90 con 0x7f9de00f6510 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.953+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f9de01072b0 con 0x7f9de00f6510 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.954+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f9de0045ad0 con 0x7f9de00f6510 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.954+0000 7f9dd5ffb700 1 -- 172.21.15.194:0/2949915694 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f9dd00095f0 con 0x7f9de00f6510 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.954+0000 7f9dd5ffb700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9dcc02ff60 0x7f9dcc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.954+0000 7f9dd5ffb700 1 -- 172.21.15.194:0/2949915694 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f9dd00410c0 con 0x7f9de00f6510 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.954+0000 7f9de48a3700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9dcc02ff60 0x7f9dcc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.955+0000 7f9de48a3700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9dcc02ff60 0x7f9dcc032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9ddc0096a0 tx=0x7f9ddc006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:54.954 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:54.959+0000 7f9dd5ffb700 1 -- 172.21.15.194:0/2949915694 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f9dd0014420 con 0x7f9de00f6510 2024-01-27T23:33:55.161 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.166+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f9de00fbf60 con 0x7f9de00f6510 2024-01-27T23:33:55.162 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.167+0000 7f9dd5ffb700 1 -- 172.21.15.194:0/2949915694 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f9dd001d7f0 con 0x7f9de00f6510 2024-01-27T23:33:55.162 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:55.162 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:55.163 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9dcc02ff60 msgr2=0x7f9dcc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9dcc02ff60 0x7f9dcc032420 secure :-1 s=READY pgs=186 cs=0 l=1 rev1=1 rx=0x7f9ddc0096a0 tx=0x7f9ddc006b40).stop 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 msgr2=0x7f9de0106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de0106730 secure :-1 s=READY pgs=440 cs=0 l=1 rev1=1 rx=0x7f9dd0013510 tx=0x7f9dd000b040).stop 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 shutdown_connections 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f9dcc02ff60 0x7f9dcc032420 unknown :-1 s=CLOSED pgs=186 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 --2- 172.21.15.194:0/2949915694 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f9de00f6510 0x7f9de0106730 unknown :-1 s=CLOSED pgs=440 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.168+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 >> 172.21.15.194:0/2949915694 conn(0x7f9de00f1c70 msgr2=0x7f9de00f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.169+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 shutdown_connections 2024-01-27T23:33:55.164 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:55.169+0000 7f9de7308700 1 -- 172.21.15.194:0/2949915694 wait complete. 2024-01-27T23:33:55.165 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:55.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:55 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:55.168158+0000 mon.smithi121 (mon.0) 318 : audit [DBG] from='client.? 172.21.15.194:0/2949915694' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:56.526 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:56.526 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:33:56.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:56 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:55.338677+0000 mgr.smithi121.hcppvz (mgr.14162) 267 : cluster [DBG] pgmap v253: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:57.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.678+0000 7f5618877700 1 -- 172.21.15.194:0/257700234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 msgr2=0x7f56140f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.678+0000 7f5618877700 1 --2- 172.21.15.194:0/257700234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f56140f5830 secure :-1 s=READY pgs=441 cs=0 l=1 rev1=1 rx=0x7f5604009fb0 tx=0x7f5604009b30).stop 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.678+0000 7f5618877700 1 -- 172.21.15.194:0/257700234 shutdown_connections 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.678+0000 7f5618877700 1 --2- 172.21.15.194:0/257700234 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f56140f5830 unknown :-1 s=CLOSED pgs=441 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.678+0000 7f5618877700 1 -- 172.21.15.194:0/257700234 >> 172.21.15.194:0/257700234 conn(0x7f56140f0b70 msgr2=0x7f56140f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.678+0000 7f5618877700 1 -- 172.21.15.194:0/257700234 shutdown_connections 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.678+0000 7f5618877700 1 -- 172.21.15.194:0/257700234 wait complete. 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.679+0000 7f5618877700 1 Processor -- start 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.679+0000 7f5618877700 1 -- start start 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.679+0000 7f5618877700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f5614105600 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.679+0000 7f5618877700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f5614105af0 con 0x7f56140f5410 2024-01-27T23:33:57.676 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f561259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f5614105600 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f561259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f5614105600 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:57448/0 (socket says 172.21.15.194:57448) 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f561259c700 1 -- 172.21.15.194:0/463603773 learned_addr learned my addr 172.21.15.194:0/463603773 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f561259c700 1 -- 172.21.15.194:0/463603773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f5604005040 con 0x7f56140f5410 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f561259c700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f5614105600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f5604013510 tx=0x7f560400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f5602ffd700 1 -- 172.21.15.194:0/463603773 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5604009320 con 0x7f56140f5410 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f5614105cf0 con 0x7f56140f5410 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f5602ffd700 1 -- 172.21.15.194:0/463603773 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f5604013f00 con 0x7f56140f5410 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f5602ffd700 1 -- 172.21.15.194:0/463603773 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f5604014d90 con 0x7f56140f5410 2024-01-27T23:33:57.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.680+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f5614106180 con 0x7f56140f5410 2024-01-27T23:33:57.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.681+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f5614045ad0 con 0x7f56140f5410 2024-01-27T23:33:57.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.681+0000 7f5602ffd700 1 -- 172.21.15.194:0/463603773 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f56040095f0 con 0x7f56140f5410 2024-01-27T23:33:57.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.681+0000 7f5602ffd700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f55fc02ff60 0x7f55fc032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:33:57.678 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.681+0000 7f5602ffd700 1 -- 172.21.15.194:0/463603773 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f5604040fe0 con 0x7f56140f5410 2024-01-27T23:33:57.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.682+0000 7f5611d9b700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f55fc02ff60 0x7f55fc032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:33:57.679 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.682+0000 7f5611d9b700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f55fc02ff60 0x7f55fc032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f56080096a0 tx=0x7f5608006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:33:57.681 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.686+0000 7f5602ffd700 1 -- 172.21.15.194:0/463603773 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f560401d7f0 con 0x7f56140f5410 2024-01-27T23:33:57.887 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.892+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f56140fae60 con 0x7f56140f5410 2024-01-27T23:33:57.888 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.892+0000 7f5602ffd700 1 -- 172.21.15.194:0/463603773 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f5604010230 con 0x7f56140f5410 2024-01-27T23:33:57.888 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:33:57.888 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:33:57.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f55fc02ff60 msgr2=0x7f55fc032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:57.889 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f55fc02ff60 0x7f55fc032420 secure :-1 s=READY pgs=187 cs=0 l=1 rev1=1 rx=0x7f56080096a0 tx=0x7f5608006b40).stop 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 msgr2=0x7f5614105600 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f5614105600 secure :-1 s=READY pgs=442 cs=0 l=1 rev1=1 rx=0x7f5604013510 tx=0x7f560400b040).stop 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 shutdown_connections 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f55fc02ff60 0x7f55fc032420 unknown :-1 s=CLOSED pgs=187 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 --2- 172.21.15.194:0/463603773 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f56140f5410 0x7f5614105600 unknown :-1 s=CLOSED pgs=442 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 >> 172.21.15.194:0/463603773 conn(0x7f56140f0b70 msgr2=0x7f56140f1630 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 shutdown_connections 2024-01-27T23:33:57.890 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:33:57.894+0000 7f5618877700 1 -- 172.21.15.194:0/463603773 wait complete. 2024-01-27T23:33:57.891 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:33:58.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:57.338931+0000 mgr.smithi121.hcppvz (mgr.14162) 268 : cluster [DBG] pgmap v254: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:33:58.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:33:58 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:33:57.893908+0000 mon.smithi121 (mon.0) 319 : audit [DBG] from='client.? 172.21.15.194:0/463603773' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:33:59.257 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:33:59.257 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:34:00.429 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.433+0000 7f7709536700 1 -- 172.21.15.194:0/241169763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 msgr2=0x7f77040f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.433+0000 7f7709536700 1 --2- 172.21.15.194:0/241169763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f77040f6930 secure :-1 s=READY pgs=443 cs=0 l=1 rev1=1 rx=0x7f76f8009fb0 tx=0x7f76f8009b30).stop 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.433+0000 7f7709536700 1 -- 172.21.15.194:0/241169763 shutdown_connections 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.433+0000 7f7709536700 1 --2- 172.21.15.194:0/241169763 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f77040f6930 unknown :-1 s=CLOSED pgs=443 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.433+0000 7f7709536700 1 -- 172.21.15.194:0/241169763 >> 172.21.15.194:0/241169763 conn(0x7f77040f1c70 msgr2=0x7f77040f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.433+0000 7f7709536700 1 -- 172.21.15.194:0/241169763 shutdown_connections 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.433+0000 7f7709536700 1 -- 172.21.15.194:0/241169763 wait complete. 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.434+0000 7f7709536700 1 Processor -- start 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.434+0000 7f7709536700 1 -- start start 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.434+0000 7f7709536700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f7704106730 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.434+0000 7f7709536700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f7704106c20 con 0x7f77040f6510 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.434+0000 7f7702ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f7704106730 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.434+0000 7f7702ffd700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f7704106730 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:57472/0 (socket says 172.21.15.194:57472) 2024-01-27T23:34:00.430 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.434+0000 7f7702ffd700 1 -- 172.21.15.194:0/1401116291 learned_addr learned my addr 172.21.15.194:0/1401116291 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.435+0000 7f7702ffd700 1 -- 172.21.15.194:0/1401116291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f76f8005040 con 0x7f77040f6510 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.435+0000 7f7702ffd700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f7704106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f76f8014000 tx=0x7f76f800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.435+0000 7f76eb7fe700 1 -- 172.21.15.194:0/1401116291 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f80095f0 con 0x7f77040f6510 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.435+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f7704106e20 con 0x7f77040f6510 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.435+0000 7f76eb7fe700 1 -- 172.21.15.194:0/1401116291 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f76f8007e70 con 0x7f77040f6510 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.435+0000 7f76eb7fe700 1 -- 172.21.15.194:0/1401116291 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f76f8014ea0 con 0x7f77040f6510 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.435+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f77041072b0 con 0x7f77040f6510 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.436+0000 7f76eb7fe700 1 -- 172.21.15.194:0/1401116291 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f76f8025070 con 0x7f77040f6510 2024-01-27T23:34:00.432 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.436+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f7704045ad0 con 0x7f77040f6510 2024-01-27T23:34:00.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.436+0000 7f76eb7fe700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76ec02ff60 0x7f76ec032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:00.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.436+0000 7f76eb7fe700 1 -- 172.21.15.194:0/1401116291 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f76f8040970 con 0x7f77040f6510 2024-01-27T23:34:00.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.436+0000 7f77027fc700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76ec02ff60 0x7f76ec032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:00.433 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.437+0000 7f77027fc700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76ec02ff60 0x7f76ec032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f76f40096a0 tx=0x7f76f4006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:00.435 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.441+0000 7f76eb7fe700 1 -- 172.21.15.194:0/1401116291 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f76f80079c0 con 0x7f77040f6510 2024-01-27T23:34:00.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.647+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f77040fbf60 con 0x7f77040f6510 2024-01-27T23:34:00.642 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.647+0000 7f76eb7fe700 1 -- 172.21.15.194:0/1401116291 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f76f800f300 con 0x7f77040f6510 2024-01-27T23:34:00.643 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:34:00.643 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:34:00.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76ec02ff60 msgr2=0x7f76ec032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:00.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76ec02ff60 0x7f76ec032420 secure :-1 s=READY pgs=188 cs=0 l=1 rev1=1 rx=0x7f76f40096a0 tx=0x7f76f4006b40).stop 2024-01-27T23:34:00.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 msgr2=0x7f7704106730 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:00.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f7704106730 secure :-1 s=READY pgs=444 cs=0 l=1 rev1=1 rx=0x7f76f8014000 tx=0x7f76f800b040).stop 2024-01-27T23:34:00.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 shutdown_connections 2024-01-27T23:34:00.644 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f76ec02ff60 0x7f76ec032420 unknown :-1 s=CLOSED pgs=188 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:00.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 --2- 172.21.15.194:0/1401116291 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f77040f6510 0x7f7704106730 unknown :-1 s=CLOSED pgs=444 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:00.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 >> 172.21.15.194:0/1401116291 conn(0x7f77040f1c70 msgr2=0x7f77040f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:00.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 shutdown_connections 2024-01-27T23:34:00.645 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:00.649+0000 7f7709536700 1 -- 172.21.15.194:0/1401116291 wait complete. 2024-01-27T23:34:00.645 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:34:00.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:00 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:33:59.339301+0000 mgr.smithi121.hcppvz (mgr.14162) 269 : cluster [DBG] pgmap v255: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:01.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:01 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:34:00.648650+0000 mon.smithi121 (mon.0) 320 : audit [DBG] from='client.? 172.21.15.194:0/1401116291' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:34:02.016 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:34:02.016 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:34:02.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:02 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:34:01.339595+0000 mgr.smithi121.hcppvz (mgr.14162) 270 : cluster [DBG] pgmap v256: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:03.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.170+0000 7f68ebd8a700 1 -- 172.21.15.194:0/3307433752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 msgr2=0x7f68e40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:03.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.170+0000 7f68ebd8a700 1 --2- 172.21.15.194:0/3307433752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40f5830 secure :-1 s=READY pgs=445 cs=0 l=1 rev1=1 rx=0x7f68d8009fb0 tx=0x7f68d8009b30).stop 2024-01-27T23:34:03.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.170+0000 7f68ebd8a700 1 -- 172.21.15.194:0/3307433752 shutdown_connections 2024-01-27T23:34:03.167 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.170+0000 7f68ebd8a700 1 --2- 172.21.15.194:0/3307433752 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40f5830 unknown :-1 s=CLOSED pgs=445 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.170+0000 7f68ebd8a700 1 -- 172.21.15.194:0/3307433752 >> 172.21.15.194:0/3307433752 conn(0x7f68e40f0b70 msgr2=0x7f68e40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.170+0000 7f68ebd8a700 1 -- 172.21.15.194:0/3307433752 shutdown_connections 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.170+0000 7f68ebd8a700 1 -- 172.21.15.194:0/3307433752 wait complete. 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68ebd8a700 1 Processor -- start 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68ebd8a700 1 -- start start 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68ebd8a700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40fd3a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68ebd8a700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f68e40fb820 con 0x7f68e40f5410 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68e9b26700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40fd3a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68e9b26700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40fd3a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:57490/0 (socket says 172.21.15.194:57490) 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68e9b26700 1 -- 172.21.15.194:0/2678846174 learned_addr learned my addr 172.21.15.194:0/2678846174 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:34:03.168 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.171+0000 7f68e9b26700 1 -- 172.21.15.194:0/2678846174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f68d8005040 con 0x7f68e40f5410 2024-01-27T23:34:03.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.172+0000 7f68e9b26700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40fd3a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f68d8013510 tx=0x7f68d800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:03.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.172+0000 7f68d67fc700 1 -- 172.21.15.194:0/2678846174 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d8009320 con 0x7f68e40f5410 2024-01-27T23:34:03.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.172+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f68e40fb9c0 con 0x7f68e40f5410 2024-01-27T23:34:03.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.172+0000 7f68d67fc700 1 -- 172.21.15.194:0/2678846174 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f68d8013f00 con 0x7f68e40f5410 2024-01-27T23:34:03.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.172+0000 7f68d67fc700 1 -- 172.21.15.194:0/2678846174 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f68d8014d90 con 0x7f68e40f5410 2024-01-27T23:34:03.169 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.172+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f68e40fbe50 con 0x7f68e40f5410 2024-01-27T23:34:03.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.173+0000 7f68d67fc700 1 -- 172.21.15.194:0/2678846174 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f68d80095f0 con 0x7f68e40f5410 2024-01-27T23:34:03.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.173+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f68e4059100 con 0x7f68e40f5410 2024-01-27T23:34:03.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.173+0000 7f68d67fc700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f68d002ff60 0x7f68d0032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:03.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.173+0000 7f68d67fc700 1 -- 172.21.15.194:0/2678846174 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f68d80410c0 con 0x7f68e40f5410 2024-01-27T23:34:03.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.173+0000 7f68e9325700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f68d002ff60 0x7f68d0032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:03.170 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.173+0000 7f68e9325700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f68d002ff60 0x7f68d0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f68e00096a0 tx=0x7f68e0006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:03.172 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.177+0000 7f68d67fc700 1 -- 172.21.15.194:0/2678846174 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f68d8014420 con 0x7f68e40f5410 2024-01-27T23:34:03.382 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.387+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f68e40fae60 con 0x7f68e40f5410 2024-01-27T23:34:03.383 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.388+0000 7f68d67fc700 1 -- 172.21.15.194:0/2678846174 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f68d8010230 con 0x7f68e40f5410 2024-01-27T23:34:03.383 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:34:03.383 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f68d002ff60 msgr2=0x7f68d0032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f68d002ff60 0x7f68d0032420 secure :-1 s=READY pgs=189 cs=0 l=1 rev1=1 rx=0x7f68e00096a0 tx=0x7f68e0006b40).stop 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 msgr2=0x7f68e40fd3a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40fd3a0 secure :-1 s=READY pgs=446 cs=0 l=1 rev1=1 rx=0x7f68d8013510 tx=0x7f68d800b040).stop 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 shutdown_connections 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f68d002ff60 0x7f68d0032420 unknown :-1 s=CLOSED pgs=189 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 --2- 172.21.15.194:0/2678846174 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f68e40f5410 0x7f68e40fd3a0 unknown :-1 s=CLOSED pgs=446 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 >> 172.21.15.194:0/2678846174 conn(0x7f68e40f0b70 msgr2=0x7f68e405fe60 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.389+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 shutdown_connections 2024-01-27T23:34:03.385 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:03.390+0000 7f68ebd8a700 1 -- 172.21.15.194:0/2678846174 wait complete. 2024-01-27T23:34:03.386 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:34:03.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:03 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:34:03.389029+0000 mon.smithi121 (mon.0) 321 : audit [DBG] from='client.? 172.21.15.194:0/2678846174' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:34:04.762 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:34:04.762 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:34:04.843 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:04 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:34:03.339947+0000 mgr.smithi121.hcppvz (mgr.14162) 271 : cluster [DBG] pgmap v257: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:05.913 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.918+0000 7f06aa77c700 1 -- 172.21.15.194:0/3328244210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 msgr2=0x7f06a40f5830 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.918+0000 7f06aa77c700 1 --2- 172.21.15.194:0/3328244210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a40f5830 secure :-1 s=READY pgs=447 cs=0 l=1 rev1=1 rx=0x7f0698009fb0 tx=0x7f0698009b30).stop 2024-01-27T23:34:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.918+0000 7f06aa77c700 1 -- 172.21.15.194:0/3328244210 shutdown_connections 2024-01-27T23:34:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.918+0000 7f06aa77c700 1 --2- 172.21.15.194:0/3328244210 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a40f5830 unknown :-1 s=CLOSED pgs=447 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.918+0000 7f06aa77c700 1 -- 172.21.15.194:0/3328244210 >> 172.21.15.194:0/3328244210 conn(0x7f06a40f0b70 msgr2=0x7f06a40f2f90 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.918+0000 7f06aa77c700 1 -- 172.21.15.194:0/3328244210 shutdown_connections 2024-01-27T23:34:05.914 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.918+0000 7f06aa77c700 1 -- 172.21.15.194:0/3328244210 wait complete. 2024-01-27T23:34:05.915 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.919+0000 7f06aa77c700 1 Processor -- start 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.919+0000 7f06aa77c700 1 -- start start 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.919+0000 7f06aa77c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a41055d0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.919+0000 7f06aa77c700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f06a4105ac0 con 0x7f06a40f5410 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.919+0000 7f06a3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a41055d0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.919+0000 7f06a3fff700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a41055d0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52360/0 (socket says 172.21.15.194:52360) 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.919+0000 7f06a3fff700 1 -- 172.21.15.194:0/2950616481 learned_addr learned my addr 172.21.15.194:0/2950616481 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.920+0000 7f06a3fff700 1 -- 172.21.15.194:0/2950616481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f0698005040 con 0x7f06a40f5410 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.920+0000 7f06a3fff700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a41055d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f0698014000 tx=0x7f069800b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.920+0000 7f06a0ff9700 1 -- 172.21.15.194:0/2950616481 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f06980095f0 con 0x7f06a40f5410 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.920+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f06a4105cc0 con 0x7f06a40f5410 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.920+0000 7f06a0ff9700 1 -- 172.21.15.194:0/2950616481 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f0698007e70 con 0x7f06a40f5410 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.920+0000 7f06a0ff9700 1 -- 172.21.15.194:0/2950616481 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f0698014ea0 con 0x7f06a40f5410 2024-01-27T23:34:05.916 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.920+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f06a4106150 con 0x7f06a40f5410 2024-01-27T23:34:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.921+0000 7f06a0ff9700 1 -- 172.21.15.194:0/2950616481 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f06980079c0 con 0x7f06a40f5410 2024-01-27T23:34:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.921+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f06a4045ad0 con 0x7f06a40f5410 2024-01-27T23:34:05.917 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.921+0000 7f06a0ff9700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f068c030000 0x7f068c0324c0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:05.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.921+0000 7f06a0ff9700 1 -- 172.21.15.194:0/2950616481 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f06980408e0 con 0x7f06a40f5410 2024-01-27T23:34:05.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.921+0000 7f06a37fe700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f068c030000 0x7f068c0324c0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:05.918 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.922+0000 7f06a37fe700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f068c030000 0x7f068c0324c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f06940096a0 tx=0x7f0694006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:05.921 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:05.926+0000 7f06a0ff9700 1 -- 172.21.15.194:0/2950616481 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f069800f950 con 0x7f06a40f5410 2024-01-27T23:34:06.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.131+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f06a4107000 con 0x7f06a40f5410 2024-01-27T23:34:06.126 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.131+0000 7f06a0ff9700 1 -- 172.21.15.194:0/2950616481 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f0698010230 con 0x7f06a40f5410 2024-01-27T23:34:06.127 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:34:06.127 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:34:06.128 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f068c030000 msgr2=0x7f068c0324c0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f068c030000 0x7f068c0324c0 secure :-1 s=READY pgs=190 cs=0 l=1 rev1=1 rx=0x7f06940096a0 tx=0x7f0694006b40).stop 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 msgr2=0x7f06a41055d0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a41055d0 secure :-1 s=READY pgs=448 cs=0 l=1 rev1=1 rx=0x7f0698014000 tx=0x7f069800b040).stop 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 shutdown_connections 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f068c030000 0x7f068c0324c0 unknown :-1 s=CLOSED pgs=190 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 --2- 172.21.15.194:0/2950616481 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f06a40f5410 0x7f06a41055d0 unknown :-1 s=CLOSED pgs=448 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 >> 172.21.15.194:0/2950616481 conn(0x7f06a40f0b70 msgr2=0x7f06a40f1610 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 shutdown_connections 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:06.133+0000 7f06aa77c700 1 -- 172.21.15.194:0/2950616481 wait complete. 2024-01-27T23:34:06.129 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:34:06.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:34:05.340303+0000 mgr.smithi121.hcppvz (mgr.14162) 272 : cluster [DBG] pgmap v258: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:06.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:06 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:34:06.132660+0000 mon.smithi121 (mon.0) 322 : audit [DBG] from='client.? 172.21.15.194:0/2950616481' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:34:07.531 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:34:07.531 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:34:08.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.693+0000 7fa52855f700 1 -- 172.21.15.194:0/314466619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 msgr2=0x7fa5200f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:08.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.693+0000 7fa52855f700 1 --2- 172.21.15.194:0/314466619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa5200f6930 secure :-1 s=READY pgs=449 cs=0 l=1 rev1=1 rx=0x7fa51c009fb0 tx=0x7fa51c009b30).stop 2024-01-27T23:34:08.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.693+0000 7fa52855f700 1 -- 172.21.15.194:0/314466619 shutdown_connections 2024-01-27T23:34:08.690 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.693+0000 7fa52855f700 1 --2- 172.21.15.194:0/314466619 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa5200f6930 unknown :-1 s=CLOSED pgs=449 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.693+0000 7fa52855f700 1 -- 172.21.15.194:0/314466619 >> 172.21.15.194:0/314466619 conn(0x7fa5200f1c70 msgr2=0x7fa5200f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.693+0000 7fa52855f700 1 -- 172.21.15.194:0/314466619 shutdown_connections 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.693+0000 7fa52855f700 1 -- 172.21.15.194:0/314466619 wait complete. 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa52855f700 1 Processor -- start 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa52855f700 1 -- start start 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa52855f700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa520108930 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa52855f700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa520108e20 con 0x7fa5200f6510 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa5262fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa520108930 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa5262fb700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa520108930 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52378/0 (socket says 172.21.15.194:52378) 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa5262fb700 1 -- 172.21.15.194:0/2509613520 learned_addr learned my addr 172.21.15.194:0/2509613520 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:34:08.691 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa5262fb700 1 -- 172.21.15.194:0/2509613520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa51c005040 con 0x7fa5200f6510 2024-01-27T23:34:08.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa5262fb700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa520108930 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa51c013510 tx=0x7fa51c00b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:08.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.694+0000 7fa516ffd700 1 -- 172.21.15.194:0/2509613520 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51c009320 con 0x7fa5200f6510 2024-01-27T23:34:08.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.695+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa520109020 con 0x7fa5200f6510 2024-01-27T23:34:08.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.695+0000 7fa516ffd700 1 -- 172.21.15.194:0/2509613520 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa51c013f00 con 0x7fa5200f6510 2024-01-27T23:34:08.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.695+0000 7fa516ffd700 1 -- 172.21.15.194:0/2509613520 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa51c014d90 con 0x7fa5200f6510 2024-01-27T23:34:08.692 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.695+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa5201094b0 con 0x7fa5200f6510 2024-01-27T23:34:08.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.696+0000 7fa516ffd700 1 -- 172.21.15.194:0/2509613520 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa51c0095f0 con 0x7fa5200f6510 2024-01-27T23:34:08.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.696+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa520045ad0 con 0x7fa5200f6510 2024-01-27T23:34:08.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.696+0000 7fa516ffd700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa50c02ffb0 0x7fa50c032470 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:08.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.696+0000 7fa516ffd700 1 -- 172.21.15.194:0/2509613520 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa51c0410c0 con 0x7fa5200f6510 2024-01-27T23:34:08.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.696+0000 7fa525afa700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa50c02ffb0 0x7fa50c032470 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:08.693 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.696+0000 7fa525afa700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa50c02ffb0 0x7fa50c032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa5100096a0 tx=0x7fa510006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:08.695 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.700+0000 7fa516ffd700 1 -- 172.21.15.194:0/2509613520 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa51c014420 con 0x7fa5200f6510 2024-01-27T23:34:08.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:08 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:34:07.340667+0000 mgr.smithi121.hcppvz (mgr.14162) 273 : cluster [DBG] pgmap v259: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:08.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.909+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa5200fbf60 con 0x7fa5200f6510 2024-01-27T23:34:08.904 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.909+0000 7fa516ffd700 1 -- 172.21.15.194:0/2509613520 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa51c010230 con 0x7fa5200f6510 2024-01-27T23:34:08.905 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:34:08.905 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:34:08.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa50c02ffb0 msgr2=0x7fa50c032470 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:08.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa50c02ffb0 0x7fa50c032470 secure :-1 s=READY pgs=191 cs=0 l=1 rev1=1 rx=0x7fa5100096a0 tx=0x7fa510006b40).stop 2024-01-27T23:34:08.906 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 msgr2=0x7fa520108930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa520108930 secure :-1 s=READY pgs=450 cs=0 l=1 rev1=1 rx=0x7fa51c013510 tx=0x7fa51c00b040).stop 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 shutdown_connections 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa50c02ffb0 0x7fa50c032470 unknown :-1 s=CLOSED pgs=191 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 --2- 172.21.15.194:0/2509613520 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5200f6510 0x7fa520108930 unknown :-1 s=CLOSED pgs=450 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 >> 172.21.15.194:0/2509613520 conn(0x7fa5200f1c70 msgr2=0x7fa5200f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 shutdown_connections 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:08.911+0000 7fa52855f700 1 -- 172.21.15.194:0/2509613520 wait complete. 2024-01-27T23:34:08.907 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:34:09.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:09 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:34:08.910648+0000 mon.smithi121 (mon.0) 323 : audit [DBG] from='client.? 172.21.15.194:0/2509613520' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:34:10.277 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:34:10.277 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:34:10.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:10 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:34:09.341022+0000 mgr.smithi121.hcppvz (mgr.14162) 274 : cluster [DBG] pgmap v260: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:11.461 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.464+0000 7fa56359e700 1 -- 172.21.15.194:0/3195969972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 msgr2=0x7fa5640f7d00 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.464+0000 7fa56359e700 1 --2- 172.21.15.194:0/3195969972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5640f7d00 secure :-1 s=READY pgs=451 cs=0 l=1 rev1=1 rx=0x7fa554009fb0 tx=0x7fa554009b30).stop 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 -- 172.21.15.194:0/3195969972 shutdown_connections 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 --2- 172.21.15.194:0/3195969972 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5640f7d00 unknown :-1 s=CLOSED pgs=451 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 -- 172.21.15.194:0/3195969972 >> 172.21.15.194:0/3195969972 conn(0x7fa5640f0120 msgr2=0x7fa5640f2540 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 -- 172.21.15.194:0/3195969972 shutdown_connections 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 -- 172.21.15.194:0/3195969972 wait complete. 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 Processor -- start 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 -- start start 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5641064e0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56359e700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7fa5641069d0 con 0x7fa5640f78e0 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5641064e0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:11.462 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56259c700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5641064e0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52392/0 (socket says 172.21.15.194:52392) 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.465+0000 7fa56259c700 1 -- 172.21.15.194:0/278043447 learned_addr learned my addr 172.21.15.194:0/278043447 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.466+0000 7fa56259c700 1 -- 172.21.15.194:0/278043447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7fa554005040 con 0x7fa5640f78e0 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.466+0000 7fa56259c700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5641064e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fa554013510 tx=0x7fa55400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.466+0000 7fa55affd700 1 -- 172.21.15.194:0/278043447 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa554009320 con 0x7fa5640f78e0 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.466+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7fa564106bd0 con 0x7fa5640f78e0 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.466+0000 7fa55affd700 1 -- 172.21.15.194:0/278043447 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7fa554013f00 con 0x7fa5640f78e0 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.466+0000 7fa55affd700 1 -- 172.21.15.194:0/278043447 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7fa554014dc0 con 0x7fa5640f78e0 2024-01-27T23:34:11.463 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.466+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7fa564107060 con 0x7fa5640f78e0 2024-01-27T23:34:11.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.467+0000 7fa55affd700 1 -- 172.21.15.194:0/278043447 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7fa5540095f0 con 0x7fa5640f78e0 2024-01-27T23:34:11.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.467+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7fa564045ad0 con 0x7fa5640f78e0 2024-01-27T23:34:11.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.467+0000 7fa55affd700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa55002ff60 0x7fa550032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:11.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.467+0000 7fa55affd700 1 -- 172.21.15.194:0/278043447 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7fa5540410c0 con 0x7fa5640f78e0 2024-01-27T23:34:11.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.467+0000 7fa561d9b700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa55002ff60 0x7fa550032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:11.464 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.468+0000 7fa561d9b700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa55002ff60 0x7fa550032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa54c0096a0 tx=0x7fa54c006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:11.467 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.472+0000 7fa55affd700 1 -- 172.21.15.194:0/278043447 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7fa554014420 con 0x7fa5640f78e0 2024-01-27T23:34:11.674 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.679+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7fa564107f10 con 0x7fa5640f78e0 2024-01-27T23:34:11.675 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.680+0000 7fa55affd700 1 -- 172.21.15.194:0/278043447 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7fa554010230 con 0x7fa5640f78e0 2024-01-27T23:34:11.675 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:34:11.675 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa55002ff60 msgr2=0x7fa550032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa55002ff60 0x7fa550032420 secure :-1 s=READY pgs=192 cs=0 l=1 rev1=1 rx=0x7fa54c0096a0 tx=0x7fa54c006b40).stop 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 msgr2=0x7fa5641064e0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5641064e0 secure :-1 s=READY pgs=452 cs=0 l=1 rev1=1 rx=0x7fa554013510 tx=0x7fa55400b040).stop 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 shutdown_connections 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7fa55002ff60 0x7fa550032420 unknown :-1 s=CLOSED pgs=192 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 --2- 172.21.15.194:0/278043447 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7fa5640f78e0 0x7fa5641064e0 unknown :-1 s=CLOSED pgs=452 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 >> 172.21.15.194:0/278043447 conn(0x7fa5640f0120 msgr2=0x7fa5640f0bc0 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 shutdown_connections 2024-01-27T23:34:11.677 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:11.681+0000 7fa56359e700 1 -- 172.21.15.194:0/278043447 wait complete. 2024-01-27T23:34:11.678 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:34:12.592 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:34:11.341401+0000 mgr.smithi121.hcppvz (mgr.14162) 275 : cluster [DBG] pgmap v261: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:12.593 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:12 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: audit 2024-01-27T23:34:11.681267+0000 mon.smithi121 (mon.0) 324 : audit [DBG] from='client.? 172.21.15.194:0/278043447' entity='client.admin' cmd=[{"prefix": "mon dump", "format": "json"}]: dispatch 2024-01-27T23:34:13.063 INFO:tasks.cephadm:Waiting for 2 mons in monmap... 2024-01-27T23:34:13.064 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm --image docker.io/ceph/ceph:v16.2.4 shell -c /etc/ceph/ceph.conf -k /etc/ceph/ceph.client.admin.keyring --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 -- ceph mon dump -f json 2024-01-27T23:34:14.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.239+0000 7f6485087700 1 -- 172.21.15.194:0/2633235243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 msgr2=0x7f64800f6930 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:14.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.239+0000 7f6485087700 1 --2- 172.21.15.194:0/2633235243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f64800f6930 secure :-1 s=READY pgs=453 cs=0 l=1 rev1=1 rx=0x7f6474009fb0 tx=0x7f6474009b30).stop 2024-01-27T23:34:14.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.239+0000 7f6485087700 1 -- 172.21.15.194:0/2633235243 shutdown_connections 2024-01-27T23:34:14.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.239+0000 7f6485087700 1 --2- 172.21.15.194:0/2633235243 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f64800f6930 unknown :-1 s=CLOSED pgs=453 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:14.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.239+0000 7f6485087700 1 -- 172.21.15.194:0/2633235243 >> 172.21.15.194:0/2633235243 conn(0x7f64800f1c70 msgr2=0x7f64800f4090 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:14.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.239+0000 7f6485087700 1 -- 172.21.15.194:0/2633235243 shutdown_connections 2024-01-27T23:34:14.235 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.239+0000 7f6485087700 1 -- 172.21.15.194:0/2633235243 wait complete. 2024-01-27T23:34:14.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f6485087700 1 Processor -- start 2024-01-27T23:34:14.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f6485087700 1 -- start start 2024-01-27T23:34:14.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f6485087700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f648006b9a0 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:14.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f6485087700 1 -- --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_getmap magic: 0 v1 -- 0x7f6480069e20 con 0x7f64800f6510 2024-01-27T23:34:14.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f647ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f648006b9a0 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:14.237 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f647ed9d700 1 --2- >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f648006b9a0 unknown :-1 s=HELLO_CONNECTING pgs=0 cs=0 l=1 rev1=1 rx=0 tx=0).handle_hello peer v2:172.21.15.121:3300/0 says I am v2:172.21.15.194:52406/0 (socket says 172.21.15.194:52406) 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f647ed9d700 1 -- 172.21.15.194:0/3583505287 learned_addr learned my addr 172.21.15.194:0/3583505287 (peer_addr_for_me v2:172.21.15.194:0/0) 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.240+0000 7f647ed9d700 1 -- 172.21.15.194:0/3583505287 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({config=0+,monmap=0+}) v3 -- 0x7f6474005040 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.241+0000 7f647ed9d700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f648006b9a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f6474013510 tx=0x7f647400b040).ready entity=mon.0 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.241+0000 7f64677fe700 1 -- 172.21.15.194:0/3583505287 <== mon.0 v2:172.21.15.121:3300/0 1 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6474009320 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.241+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({mgrmap=0+}) v3 -- 0x7f6480069fc0 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.241+0000 7f64677fe700 1 -- 172.21.15.194:0/3583505287 <== mon.0 v2:172.21.15.121:3300/0 2 ==== config(25 keys) v1 ==== 1074+0+0 (secure 0 0 0) 0x7f6474013f00 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.241+0000 7f64677fe700 1 -- 172.21.15.194:0/3583505287 <== mon.0 v2:172.21.15.121:3300/0 3 ==== mon_map magic: 0 v1 ==== 229+0+0 (secure 0 0 0) 0x7f6474014d90 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.241+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_subscribe({osdmap=0}) v3 -- 0x7f648006a450 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.242+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "get_command_descriptions"} v 0) v1 -- 0x7f6480059100 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.242+0000 7f64677fe700 1 -- 172.21.15.194:0/3583505287 <== mon.0 v2:172.21.15.121:3300/0 4 ==== mgrmap(e 17) v1 ==== 36269+0+0 (secure 0 0 0) 0x7f64740095f0 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.242+0000 7f64677fe700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f646802ff60 0x7f6468032420 unknown :-1 s=NONE pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0).connect 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.242+0000 7f64677fe700 1 -- 172.21.15.194:0/3583505287 <== mon.0 v2:172.21.15.121:3300/0 5 ==== osd_map(5..5 src has 1..5) v4 ==== 1270+0+0 (secure 0 0 0) 0x7f6474041000 con 0x7f64800f6510 2024-01-27T23:34:14.238 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.242+0000 7f647e59c700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f646802ff60 0x7f6468032420 unknown :-1 s=BANNER_CONNECTING pgs=0 cs=0 l=1 rev1=0 rx=0 tx=0)._handle_peer_banner_payload supported=1 required=0 2024-01-27T23:34:14.239 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.242+0000 7f647e59c700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f646802ff60 0x7f6468032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f64700096a0 tx=0x7f6470006b40).ready entity=mgr.14162 client_cookie=0 server_cookie=0 in_seq=0 out_seq=0 2024-01-27T23:34:14.241 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.246+0000 7f64677fe700 1 -- 172.21.15.194:0/3583505287 <== mon.0 v2:172.21.15.121:3300/0 6 ==== mon_command_ack([{"prefix": "get_command_descriptions"}]=0 v0) v1 ==== 72+0+152212 (secure 0 0 0) 0x7f647401d7f0 con 0x7f64800f6510 2024-01-27T23:34:14.447 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.452+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 --> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] -- mon_command({"prefix": "mon dump", "format": "json"} v 0) v1 -- 0x7f64800fbf60 con 0x7f64800f6510 2024-01-27T23:34:14.448 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.453+0000 7f64677fe700 1 -- 172.21.15.194:0/3583505287 <== mon.0 v2:172.21.15.121:3300/0 7 ==== mon_command_ack([{"prefix": "mon dump", "format": "json"}]=0 dumped monmap epoch 1 v1) v1 ==== 95+0+694 (secure 0 0 0) 0x7f6474010230 con 0x7f64800f6510 2024-01-27T23:34:14.448 INFO:teuthology.orchestra.run.smithi194.stdout: 2024-01-27T23:34:14.448 INFO:teuthology.orchestra.run.smithi194.stdout:{"epoch":1,"fsid":"2005e414-bd6b-11ee-95b2-87774f69a715","modified":"2024-01-27T23:24:21.255875Z","created":"2024-01-27T23:24:21.255875Z","min_mon_release":16,"min_mon_release_name":"pacific","election_strategy":1,"disallowed_leaders: ":"","stretch_mode":false,"features":{"persistent":["kraken","luminous","mimic","osdmap-prune","nautilus","octopus","pacific","elector-pinging"],"optional":[]},"mons":[{"rank":0,"name":"smithi121","public_addrs":{"addrvec":[{"type":"v2","addr":"172.21.15.121:3300","nonce":0},{"type":"v1","addr":"172.21.15.121:6789","nonce":0}]},"addr":"172.21.15.121:6789/0","public_addr":"172.21.15.121:6789/0","priority":0,"weight":0,"crush_location":"{}"}],"quorum":[0]} 2024-01-27T23:34:14.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f646802ff60 msgr2=0x7f6468032420 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:14.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f646802ff60 0x7f6468032420 secure :-1 s=READY pgs=193 cs=0 l=1 rev1=1 rx=0x7f64700096a0 tx=0x7f6470006b40).stop 2024-01-27T23:34:14.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 msgr2=0x7f648006b9a0 secure :-1 s=STATE_CONNECTION_ESTABLISHED l=1).mark_down 2024-01-27T23:34:14.450 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f648006b9a0 secure :-1 s=READY pgs=454 cs=0 l=1 rev1=1 rx=0x7f6474013510 tx=0x7f647400b040).stop 2024-01-27T23:34:14.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 shutdown_connections 2024-01-27T23:34:14.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:6800/802027800,v1:172.21.15.121:6801/802027800] conn(0x7f646802ff60 0x7f6468032420 unknown :-1 s=CLOSED pgs=193 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:14.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 --2- 172.21.15.194:0/3583505287 >> [v2:172.21.15.121:3300/0,v1:172.21.15.121:6789/0] conn(0x7f64800f6510 0x7f648006b9a0 unknown :-1 s=CLOSED pgs=454 cs=0 l=1 rev1=1 rx=0 tx=0).stop 2024-01-27T23:34:14.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 >> 172.21.15.194:0/3583505287 conn(0x7f64800f1c70 msgr2=0x7f64800f2760 unknown :-1 s=STATE_NONE l=0).mark_down 2024-01-27T23:34:14.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 shutdown_connections 2024-01-27T23:34:14.451 INFO:teuthology.orchestra.run.smithi194.stderr:2024-01-27T23:34:14.454+0000 7f6485087700 1 -- 172.21.15.194:0/3583505287 wait complete. 2024-01-27T23:34:14.451 INFO:teuthology.orchestra.run.smithi194.stderr:dumped monmap epoch 1 2024-01-27T23:34:14.842 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:14 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: cluster 2024-01-27T23:34:13.341781+0000 mgr.smithi121.hcppvz (mgr.14162) 276 : cluster [DBG] pgmap v262: 0 pgs: ; 0 B data, 0 B used, 0 B / 0 B avail 2024-01-27T23:34:14.872 ERROR:teuthology.contextutil:Saw exception from nested tasks Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds 2024-01-27T23:34:14.874 INFO:tasks.cephadm:Cleaning up testdir ceph.* files... 2024-01-27T23:34:14.874 DEBUG:teuthology.orchestra.run.smithi121:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-27T23:34:14.898 DEBUG:teuthology.orchestra.run.smithi194:> rm -f /home/ubuntu/cephtest/seed.ceph.conf /home/ubuntu/cephtest/ceph.pub 2024-01-27T23:34:14.916 INFO:tasks.cephadm:Stopping all daemons... 2024-01-27T23:34:14.917 INFO:tasks.cephadm.mon.smithi121:Stopping mon.smithi121... 2024-01-27T23:34:14.917 DEBUG:teuthology.orchestra.run.smithi121:> sudo systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121 2024-01-27T23:34:15.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:14 smithi121 systemd[1]: Stopping Ceph mon.smithi121 for 2005e414-bd6b-11ee-95b2-87774f69a715... 2024-01-27T23:34:15.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:15 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:34:15.060+0000 7f8518355700 -1 received signal: Terminated from /run/podman-init -- /usr/bin/ceph-mon -n mon.smithi121 -f --setuser ceph --setgroup ceph --default-log-to-file=false --default-log-to-stderr=true --default-log-stderr-prefix=debug --default-mon-cluster-log-to-file=false --default-mon-cluster-log-to-stderr=true (PID: 1) UID: 0 2024-01-27T23:34:15.256 INFO:journalctl@ceph.mon.smithi121.smithi121.stdout:Jan 27 23:34:15 smithi121 ceph-2005e414-bd6b-11ee-95b2-87774f69a715-mon.smithi121[38787]: debug 2024-01-27T23:34:15.060+0000 7f8518355700 -1 mon.smithi121@0(leader) e1 *** Got Signal Terminated *** 2024-01-27T23:34:15.402 DEBUG:teuthology.orchestra.run.smithi121:> sudo pkill -f 'journalctl -f -n 0 -u ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi121.service' 2024-01-27T23:34:15.468 DEBUG:teuthology.orchestra.run:got remote process result: None 2024-01-27T23:34:15.468 INFO:tasks.cephadm.mon.smithi121:Stopped mon.smithi121 2024-01-27T23:34:15.468 INFO:tasks.cephadm.mon.smithi194:Stopping mon.smithi194... 2024-01-27T23:34:15.468 DEBUG:teuthology.orchestra.run.smithi194:> sudo systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194 2024-01-27T23:34:15.501 INFO:teuthology.orchestra.run.smithi194.stderr:Failed to stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194.service: Unit ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194.service not loaded. 2024-01-27T23:34:15.502 DEBUG:teuthology.orchestra.run:got remote process result: 5 2024-01-27T23:34:15.503 ERROR:tasks.cephadm:Failed to stop "ceph.mon.smithi194" Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 5: 'sudo systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194' 2024-01-27T23:34:15.504 INFO:tasks.cephadm:Archiving crash dumps... 2024-01-27T23:34:15.506 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122/remote/ubuntu@smithi121.front.sepia.ceph.com/crash 2024-01-27T23:34:15.507 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/crash -- . 2024-01-27T23:34:15.556 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/crash to /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122/remote/ubuntu@smithi194.front.sepia.ceph.com/crash 2024-01-27T23:34:15.557 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/crash -- . 2024-01-27T23:34:15.596 INFO:teuthology.orchestra.run.smithi194.stderr:tar: /var/lib/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/crash: Cannot open: No such file or directory 2024-01-27T23:34:15.596 INFO:teuthology.orchestra.run.smithi194.stderr:tar: Error is not recoverable: exiting now 2024-01-27T23:34:15.597 INFO:tasks.cephadm:Checking cluster log for badness... 2024-01-27T23:34:15.597 DEBUG:teuthology.orchestra.run.smithi121:> sudo egrep '\[ERR\]|\[WRN\]|\[SEC\]' /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph.log | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v 'overall HEALTH_' | egrep -v '\(FS_DEGRADED\)' | egrep -v '\(MDS_FAILED\)' | egrep -v '\(MDS_DEGRADED\)' | egrep -v '\(FS_WITH_FAILED_MDS\)' | egrep -v '\(MDS_DAMAGE\)' | egrep -v '\(MDS_ALL_DOWN\)' | egrep -v '\(MDS_UP_LESS_THAN_MAX\)' | egrep -v '\(FS_INLINE_DATA_DEPRECATED\)' | egrep -v 'overall HEALTH_' | egrep -v '\(OSD_DOWN\)' | egrep -v '\(OSD_' | egrep -v 'but it is still running' | egrep -v 'is not responding' | head -n 1 2024-01-27T23:34:15.626 INFO:teuthology.orchestra.run.smithi121.stderr:grep: /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph.log: No such file or directory 2024-01-27T23:34:15.628 INFO:tasks.cephadm:Compressing logs... 2024-01-27T23:34:15.628 DEBUG:teuthology.orchestra.run.smithi121:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-27T23:34:15.671 DEBUG:teuthology.orchestra.run.smithi194:> time sudo find /var/log/ceph /var/log/rbd-target-api -name '*.log' -print0 | sudo xargs --max-args=1 --max-procs=0 --verbose -0 --no-run-if-empty -- gzip -5 --verbose -- 2024-01-27T23:34:15.695 INFO:teuthology.orchestra.run.smithi121.stderr:find: ‘/var/log/rbd-target-api’: No such file or directory 2024-01-27T23:34:15.695 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-27T23:34:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-volume.log 2024-01-27T23:34:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:gzip -5 --verbose -- /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-mgr.smithi121.hcppvz.log 2024-01-27T23:34:15.699 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/cephadm.log: gzip -5 --verbose -- /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-mon.smithi121.log 2024-01-27T23:34:15.700 INFO:teuthology.orchestra.run.smithi121.stderr:/var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-volume.log: /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-mgr.smithi121.hcppvz.log: /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-mon.smithi121.log: 90.2% -- replaced with /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-volume.log.gz 2024-01-27T23:34:15.701 INFO:teuthology.orchestra.run.smithi194.stderr:find: ‘/var/log/rbd-target-api’gzip -5 --verbose -- /var/log/ceph/cephadm.log 2024-01-27T23:34:15.701 INFO:teuthology.orchestra.run.smithi194.stderr:: No such file or directory 2024-01-27T23:34:15.701 INFO:teuthology.orchestra.run.smithi194.stderr:/var/log/ceph/cephadm.log: 96.1% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-27T23:34:15.701 INFO:teuthology.orchestra.run.smithi121.stderr: 92.2% -- replaced with /var/log/ceph/cephadm.log.gz 2024-01-27T23:34:15.702 INFO:teuthology.orchestra.run.smithi194.stderr: 2024-01-27T23:34:15.702 INFO:teuthology.orchestra.run.smithi194.stderr:real 0m0.013s 2024-01-27T23:34:15.702 INFO:teuthology.orchestra.run.smithi194.stderr:user 0m0.010s 2024-01-27T23:34:15.702 INFO:teuthology.orchestra.run.smithi194.stderr:sys 0m0.011s 2024-01-27T23:34:15.771 INFO:teuthology.orchestra.run.smithi121.stderr: 92.6% -- replaced with /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-mgr.smithi121.hcppvz.log.gz 2024-01-27T23:34:15.836 INFO:teuthology.orchestra.run.smithi121.stderr: 90.6% -- replaced with /var/log/ceph/2005e414-bd6b-11ee-95b2-87774f69a715/ceph-mon.smithi121.log.gz 2024-01-27T23:34:15.838 INFO:teuthology.orchestra.run.smithi121.stderr: 2024-01-27T23:34:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:real 0m0.152s 2024-01-27T23:34:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:user 0m0.226s 2024-01-27T23:34:15.838 INFO:teuthology.orchestra.run.smithi121.stderr:sys 0m0.016s 2024-01-27T23:34:15.838 INFO:tasks.cephadm:Archiving logs... 2024-01-27T23:34:15.839 DEBUG:teuthology.misc:Transferring archived files from smithi121:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122/remote/ubuntu@smithi121.front.sepia.ceph.com/log 2024-01-27T23:34:15.840 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-27T23:34:15.953 DEBUG:teuthology.misc:Transferring archived files from smithi194:/var/log/ceph to /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122/remote/ubuntu@smithi194.front.sepia.ceph.com/log 2024-01-27T23:34:15.953 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /var/log/ceph -- . 2024-01-27T23:34:15.982 INFO:tasks.cephadm:Removing cluster... 2024-01-27T23:34:15.982 DEBUG:teuthology.orchestra.run.smithi121:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 --force 2024-01-27T23:35:14.018 DEBUG:teuthology.orchestra.run.smithi194:> sudo /home/ubuntu/cephtest/cephadm rm-cluster --fsid 2005e414-bd6b-11ee-95b2-87774f69a715 --force 2024-01-27T23:35:14.322 INFO:tasks.cephadm:Removing cephadm ... 2024-01-27T23:35:14.322 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-27T23:35:14.341 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf /home/ubuntu/cephtest/cephadm 2024-01-27T23:35:14.358 INFO:tasks.cephadm:Teardown complete 2024-01-27T23:35:14.359 ERROR:teuthology.run_tasks:Saw exception from tasks. Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 5: 'sudo systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194' 2024-01-27T23:35:14.586 ERROR:teuthology.util.sentry: Sentry event: https://sentry.ceph.com/organizations/ceph/?query=7239b813b8194ef499a4f1878f47d4e8 Traceback (most recent call last): File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 512, in ceph_bootstrap yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 30, in nested vars.append(enter()) File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 638, in ceph_mons while proceed(): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 134, in __call__ raise MaxWhileTries(error_msg) teuthology.exceptions.MaxWhileTries: reached maximum tries (181) after waiting for 180 seconds During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/run_tasks.py", line 109, in run_tasks manager.__enter__() File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 1463, in task with contextutil.nested( File "/usr/lib/python3.8/contextlib.py", line 113, in __enter__ return next(self.gen) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 54, in nested raise exc[1] File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 1376, in initialize_config yield File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/contextutil.py", line 46, in nested if exit(*exc): File "/usr/lib/python3.8/contextlib.py", line 131, in __exit__ self.gen.throw(type, value, traceback) File "/home/teuthworker/src/github.com_ceph_ceph-c_585bdf827237450fb80df4a856eba434a64948fa/qa/tasks/cephadm.py", line 531, in ceph_bootstrap ctx.daemons.get_daemon(type_, id_, cluster).stop() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/daemon/cephadmunit.py", line 154, in stop self.remote.sh(self.stop_cmd) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 97, in sh proc = self.run(**kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/remote.py", line 523, in run r = self._runner(client=self.ssh, name=self.shortname, **kwargs) File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 455, in run r.wait() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 161, in wait self._raise_for_status() File "/home/teuthworker/src/git.ceph.com_teuthology_c856e0f9244dd29a50697123b480d63f16a4ed81/teuthology/orchestra/run.py", line 181, in _raise_for_status raise CommandFailedError( teuthology.exceptions.CommandFailedError: Command failed on smithi194 with status 5: 'sudo systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194' 2024-01-27T23:35:14.591 DEBUG:teuthology.run_tasks:Unwinding manager cephadm 2024-01-27T23:35:14.600 DEBUG:teuthology.run_tasks:Unwinding manager install 2024-01-27T23:35:14.608 INFO:teuthology.task.install.util:Removing shipped files: /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer... 2024-01-27T23:35:14.608 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-27T23:35:14.633 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /home/ubuntu/cephtest/valgrind.supp /usr/bin/daemon-helper /usr/bin/adjust-ulimits /usr/bin/stdin-killer 2024-01-27T23:35:14.703 DEBUG:teuthology.run_tasks:Unwinding manager clock 2024-01-27T23:35:14.712 INFO:teuthology.task.clock:Checking final clock skew... 2024-01-27T23:35:14.712 DEBUG:teuthology.orchestra.run.smithi121:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T23:35:14.715 DEBUG:teuthology.orchestra.run.smithi194:> PATH=/usr/bin:/usr/sbin ntpq -p || PATH=/usr/bin:/usr/sbin chronyc sources || true 2024-01-27T23:35:14.729 INFO:teuthology.orchestra.run.smithi121.stderr:bash: ntpq: command not found 2024-01-27T23:35:14.731 INFO:teuthology.orchestra.run.smithi194.stderr:bash: ntpq: command not found 2024-01-27T23:35:14.733 INFO:teuthology.orchestra.run.smithi121.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T23:35:14.733 INFO:teuthology.orchestra.run.smithi121.stdout:=============================================================================== 2024-01-27T23:35:14.733 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 39 -109us[ -91us] +/- 89ms 2024-01-27T23:35:14.734 INFO:teuthology.orchestra.run.smithi121.stdout:^* hv02.front.sepia.ceph.com 3 6 377 38 +1305us[+1323us] +/- 65ms 2024-01-27T23:35:14.734 INFO:teuthology.orchestra.run.smithi121.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 37 -1261us[-1261us] +/- 66ms 2024-01-27T23:35:14.734 INFO:teuthology.orchestra.run.smithi121.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:35:14.735 INFO:teuthology.orchestra.run.smithi194.stdout:MS Name/IP address Stratum Poll Reach LastRx Last sample 2024-01-27T23:35:14.735 INFO:teuthology.orchestra.run.smithi194.stdout:=============================================================================== 2024-01-27T23:35:14.735 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv01.front.sepia.ceph.com 2 6 377 40 -100us[ -100us] +/- 88ms 2024-01-27T23:35:14.736 INFO:teuthology.orchestra.run.smithi194.stdout:^* hv02.front.sepia.ceph.com 3 6 377 43 +1285us[+1281us] +/- 65ms 2024-01-27T23:35:14.736 INFO:teuthology.orchestra.run.smithi194.stdout:^+ hv03.front.sepia.ceph.com 3 6 377 41 -1232us[-1232us] +/- 66ms 2024-01-27T23:35:14.736 INFO:teuthology.orchestra.run.smithi194.stdout:^? hv04.front.sepia.ceph.com 0 8 0 - +0ns[ +0ns] +/- 0ns 2024-01-27T23:35:14.737 DEBUG:teuthology.run_tasks:Unwinding manager ansible.cephlab 2024-01-27T23:35:14.746 INFO:teuthology.task.ansible:Skipping ansible cleanup... 2024-01-27T23:35:14.747 DEBUG:teuthology.run_tasks:Unwinding manager selinux 2024-01-27T23:35:14.755 DEBUG:teuthology.orchestra.run.smithi121:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-27T23:35:14.836 DEBUG:teuthology.orchestra.run.smithi194:> mkdir /home/ubuntu/cephtest/archive/audit && sudo cp /var/log/audit/audit.log /home/ubuntu/cephtest/archive/audit && sudo chown $USER /home/ubuntu/cephtest/archive/audit/audit.log && gzip /home/ubuntu/cephtest/archive/audit/audit.log 2024-01-27T23:35:14.899 DEBUG:teuthology.orchestra.run.smithi121:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T23:35:14.927 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:35:14.928 DEBUG:teuthology.orchestra.run.smithi194:> sudo grep -a 'avc: .*denied' /var/log/audit/audit.log | grep -av -e 'comm="dmidecode"' -e chronyd.service -e 'name="cephtest"' -e scontext=system_u:system_r:nrpe_t:s0 -e scontext=system_u:system_r:pcp_pmlogger_t -e scontext=system_u:system_r:pcp_pmcd_t:s0 -e 'comm="rhsmd"' -e scontext=system_u:system_r:syslogd_t:s0 -e tcontext=system_u:system_r:nrpe_t:s0 -e 'comm="updatedb"' -e 'comm="smartd"' -e 'comm="rhsmcertd-worke"' -e 'comm="setroubleshootd"' -e 'comm="rpm"' -e tcontext=system_u:object_r:container_runtime_exec_t:s0 -e 'comm="ksmtuned"' -e 'comm="sssd"' -e 'comm="sss_cache"' -e context=system_u:system_r:NetworkManager_dispatcher_t:s0 2024-01-27T23:35:14.956 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:35:14.957 DEBUG:teuthology.run_tasks:Unwinding manager pcp 2024-01-27T23:35:14.967 DEBUG:teuthology.run_tasks:Unwinding manager internal.timer 2024-01-27T23:35:14.975 INFO:teuthology.task.internal:Duration was 1230.615518 seconds 2024-01-27T23:35:14.976 DEBUG:teuthology.run_tasks:Unwinding manager internal.syslog 2024-01-27T23:35:14.985 INFO:teuthology.task.internal.syslog:Shutting down syslog monitoring... 2024-01-27T23:35:14.985 DEBUG:teuthology.orchestra.run.smithi121:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-27T23:35:14.987 DEBUG:teuthology.orchestra.run.smithi194:> sudo rm -f -- /etc/rsyslog.d/80-cephtest.conf && sudo service rsyslog restart 2024-01-27T23:35:15.031 INFO:teuthology.orchestra.run.smithi121.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T23:35:15.043 INFO:teuthology.orchestra.run.smithi194.stderr:Redirecting to /bin/systemctl restart rsyslog.service 2024-01-27T23:35:15.355 INFO:teuthology.task.internal.syslog:Checking logs for errors... 2024-01-27T23:35:15.355 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi121.front.sepia.ceph.com 2024-01-27T23:35:15.355 DEBUG:teuthology.orchestra.run.smithi121:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-27T23:35:15.377 DEBUG:teuthology.task.internal.syslog:Checking ubuntu@smithi194.front.sepia.ceph.com 2024-01-27T23:35:15.377 DEBUG:teuthology.orchestra.run.smithi194:> egrep --binary-files=text '\bBUG\b|\bINFO\b|\bDEADLOCK\b' /home/ubuntu/cephtest/archive/syslog/kern.log | grep -v 'task .* blocked for more than .* seconds' | grep -v 'lockdep is turned off' | grep -v 'trying to register non-static key' | grep -v 'DEBUG: fsize' | grep -v CRON | grep -v 'BUG: bad unlock balance detected' | grep -v 'inconsistent lock state' | grep -v '*** DEADLOCK ***' | grep -v 'INFO: possible irq lock inversion dependency detected' | grep -v 'INFO: NMI handler (perf_event_nmi_handler) took too long to run' | grep -v 'INFO: recovery required on readonly' | grep -v 'ceph-create-keys: INFO' | grep -v INFO:ceph-create-keys | grep -v 'Loaded datasource DataSourceOpenStack' | grep -v 'container-storage-setup: INFO: Volume group backing root filesystem could not be determined' | egrep -v '\bsalt-master\b|\bsalt-minion\b|\bsalt-api\b' | grep -v ceph-crash | egrep -v '\btcmu-runner\b.*\bINFO\b' | head -n 1 2024-01-27T23:35:15.400 INFO:teuthology.task.internal.syslog:Compressing syslogs... 2024-01-27T23:35:15.400 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-27T23:35:15.420 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest/archive/syslog -name '*.log' -print0 | sudo xargs -0 --no-run-if-empty -- gzip -- 2024-01-27T23:35:15.489 INFO:teuthology.task.internal.syslog:Gathering journactl -b0... 2024-01-27T23:35:15.490 DEBUG:teuthology.orchestra.run.smithi121:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-27T23:35:15.492 DEBUG:teuthology.orchestra.run.smithi194:> sudo journalctl -b0 | gzip -9 > /home/ubuntu/cephtest/archive/syslog/journalctl-b0.gz 2024-01-27T23:35:15.580 DEBUG:teuthology.run_tasks:Unwinding manager internal.sudo 2024-01-27T23:35:15.591 INFO:teuthology.task.internal:Restoring /etc/sudoers... 2024-01-27T23:35:15.592 DEBUG:teuthology.orchestra.run.smithi121:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-27T23:35:15.618 DEBUG:teuthology.orchestra.run.smithi194:> sudo mv -f /etc/sudoers.orig.teuthology /etc/sudoers 2024-01-27T23:35:15.646 DEBUG:teuthology.run_tasks:Unwinding manager internal.coredump 2024-01-27T23:35:15.656 DEBUG:teuthology.orchestra.run.smithi121:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-27T23:35:15.662 DEBUG:teuthology.orchestra.run.smithi194:> sudo sysctl -w kernel.core_pattern=core && sudo bash -c 'for f in `find /home/ubuntu/cephtest/archive/coredump -type f`; do file $f | grep -q systemd-sysusers && rm $f || true ; done' && rmdir --ignore-fail-on-non-empty -- /home/ubuntu/cephtest/archive/coredump 2024-01-27T23:35:15.686 INFO:teuthology.orchestra.run.smithi121.stdout:kernel.core_pattern = core 2024-01-27T23:35:15.712 INFO:teuthology.orchestra.run.smithi194.stdout:kernel.core_pattern = core 2024-01-27T23:35:15.737 DEBUG:teuthology.orchestra.run.smithi121:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-27T23:35:15.766 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:35:15.767 DEBUG:teuthology.orchestra.run.smithi194:> test -e /home/ubuntu/cephtest/archive/coredump 2024-01-27T23:35:15.794 DEBUG:teuthology.orchestra.run:got remote process result: 1 2024-01-27T23:35:15.794 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive 2024-01-27T23:35:15.805 INFO:teuthology.task.internal:Transferring archived files... 2024-01-27T23:35:15.805 DEBUG:teuthology.misc:Transferring archived files from smithi121:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122/remote/smithi121 2024-01-27T23:35:15.806 DEBUG:teuthology.orchestra.run.smithi121:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-27T23:35:15.846 DEBUG:teuthology.misc:Transferring archived files from smithi194:/home/ubuntu/cephtest/archive to /home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122/remote/smithi194 2024-01-27T23:35:15.847 DEBUG:teuthology.orchestra.run.smithi194:> sudo tar c -f - -C /home/ubuntu/cephtest/archive -- . 2024-01-27T23:35:15.886 INFO:teuthology.task.internal:Removing archive directory... 2024-01-27T23:35:15.887 DEBUG:teuthology.orchestra.run.smithi121:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-27T23:35:15.889 DEBUG:teuthology.orchestra.run.smithi194:> rm -rf -- /home/ubuntu/cephtest/archive 2024-01-27T23:35:15.938 DEBUG:teuthology.run_tasks:Unwinding manager internal.archive_upload 2024-01-27T23:35:15.950 INFO:teuthology.task.internal:Not uploading archives. 2024-01-27T23:35:15.951 DEBUG:teuthology.run_tasks:Unwinding manager internal.base 2024-01-27T23:35:15.962 INFO:teuthology.task.internal:Tidying up after the test... 2024-01-27T23:35:15.963 DEBUG:teuthology.orchestra.run.smithi121:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-27T23:35:15.965 DEBUG:teuthology.orchestra.run.smithi194:> find /home/ubuntu/cephtest -ls ; rmdir -- /home/ubuntu/cephtest 2024-01-27T23:35:15.980 INFO:teuthology.orchestra.run.smithi121.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 27 23:35 /home/ubuntu/cephtest 2024-01-27T23:35:15.995 INFO:teuthology.orchestra.run.smithi194.stdout: 393224 4 drwxr-xr-x 2 ubuntu ubuntu 4096 Jan 27 23:35 /home/ubuntu/cephtest 2024-01-27T23:35:15.996 DEBUG:teuthology.run_tasks:Unwinding manager kernel 2024-01-27T23:35:16.007 DEBUG:teuthology.run_tasks:Unwinding manager console_log 2024-01-27T23:35:16.096 INFO:teuthology.nuke:Checking targets against current locks 2024-01-27T23:35:16.125 DEBUG:teuthology.nuke:shortname: smithi121 2024-01-27T23:35:16.125 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T23:35:16.151 DEBUG:teuthology.nuke:shortname: smithi194 2024-01-27T23:35:16.151 INFO:teuthology.task.internal.check_lock:Checking locks... 2024-01-27T23:35:16.155 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi121.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 23:06:26.016339', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBHWGgLFI/ZvCOpCwT7jcXXfQi77YaJe7dKPTApJcsX4KnN/15O0Y+/1pOo+BQ/PWu/WwXla2EVZGsEG6QKm3+S4='} 2024-01-27T23:35:16.177 DEBUG:teuthology.task.internal.check_lock:machine status is {'name': 'smithi194.front.sepia.ceph.com', 'description': '/home/teuthworker/archive/yuriw-2024-01-26_16:10:31-rados-pacific-release-distro-default-smithi/7535122', 'up': True, 'machine_type': 'smithi', 'is_vm': False, 'vm_host': None, 'os_type': 'centos', 'os_version': '8', 'arch': 'x86_64', 'locked': True, 'locked_since': '2024-01-27 23:06:26.017527', 'locked_by': 'scheduled_yuriw@teuthology', 'mac_address': None, 'ssh_pub_key': 'ecdsa-sha2-nistp256 AAAAE2VjZHNhLXNoYTItbmlzdHAyNTYAAAAIbmlzdHAyNTYAAABBBM8Nzw7Mdc6JkoonhBu9o8XjQxMVwogjIQPozN+3mETOrwYHuQVrKdGqNGS/m9MhFAmdsIFmuURupmmSkE6ENT8='} 2024-01-27T23:35:16.209 INFO:teuthology.orchestra.console.smithi121:Power off 2024-01-27T23:35:16.209 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-27T23:35:16.232 DEBUG:teuthology.orchestra.console.smithi121:power off output: Chassis Power Control: Down/Off 2024-01-27T23:35:16.233 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T23:35:16.248 INFO:teuthology.orchestra.console.smithi194:Power off 2024-01-27T23:35:16.249 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power off 2024-01-27T23:35:16.257 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-01-27T23:35:16.270 DEBUG:teuthology.orchestra.console.smithi194:power off output: Chassis Power Control: Down/Off 2024-01-27T23:35:16.270 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T23:35:16.292 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-27T23:35:20.258 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T23:35:20.293 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T23:35:20.368 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is on 2024-01-27T23:35:20.402 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-27T23:35:24.372 DEBUG:teuthology.orchestra.console.smithi121:pexpect command: ipmitool -H smithi121.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T23:35:24.402 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T23:35:24.482 DEBUG:teuthology.orchestra.console.smithi121:check power output: Chassis Power is off 2024-01-27T23:35:24.512 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is on 2024-01-27T23:35:24.584 INFO:teuthology.orchestra.console.smithi121:Power off completed 2024-01-27T23:35:28.516 DEBUG:teuthology.orchestra.console.smithi194:pexpect command: ipmitool -H smithi194.ipmi.sepia.ceph.com -I lanplus -U inktank -P ApGNXcA7 power status 2024-01-27T23:35:28.626 DEBUG:teuthology.orchestra.console.smithi194:check power output: Chassis Power is off 2024-01-27T23:35:28.727 INFO:teuthology.orchestra.console.smithi194:Power off completed 2024-01-27T23:35:28.832 INFO:teuthology.run:Summary data: description: rados/cephadm/mds_upgrade_sequence/{bluestore-bitmap centos_8.stream_container_tools conf/{client mds mon osd} overrides/{ignorelist_health ignorelist_wrongly_marked_down pg-warn} roles tasks/{0-from/v16.2.4 1-volume/{0-create 1-ranks/1 2-allow_standby_replay/yes 3-inline/no 4-verify} 2-client 3-upgrade-with-workload 4-verify}} duration: 1230.6155183315277 failure_reason: 'Command failed on smithi194 with status 5: ''sudo systemctl stop ceph-2005e414-bd6b-11ee-95b2-87774f69a715@mon.smithi194''' flavor: default owner: scheduled_yuriw@teuthology sentry_event: https://sentry.ceph.com/organizations/ceph/?query=7239b813b8194ef499a4f1878f47d4e8 status: fail success: false 2024-01-27T23:35:28.833 DEBUG:teuthology.report:Pushing job info to https://paddles.front.sepia.ceph.com/ 2024-01-27T23:35:28.906 INFO:teuthology.run:FAIL